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PREFACE 



During the past decade significant progress has been made in the area of child and adolescent 
mental health services. In 1982 Jane Knitzer documented the failure of our service delivery 
systems to provide both adequate and appropriate care to the increasing numbers of children 
with or at risk of developing serious emotional disturbances in this country. Following this 
publication and the subsequent formation of the Child and Adolescent Service System Program 
(CASSP) of the National Institute of Mental Health (NIMH), there has been a steady growth in 
the development of children's mental health services and increased utilization of the 
comprehensive, community-based "system of care" model as an alternative to traditional service 
approaches. Along with these changes, there has been increased attention on local, state and 
federal levels to legislative changes that would mandate the provision of comprehensive and 
coordinated services to children and adolescents. The formation of a strong and growing 
network of parent support groups and statewide and national parent organizations is yet 
another indication of the dramatic and positive advances that have ensued* 

Only during the past few years, however, has the need for accelerating and expanding the 
research base regarding the etiology, diagnosis and treatment of child and adolescent mental 
disorders as well as regarding the establishment and maintenance of effective service delivery 
system, become a major focus for the NIMH. This concern has culminated in the recent 
development of a National Plan for Researc h on Child and Adolescent Mental Disorders which 
was built upon the 1989 report of the Institute of Medicine of the National Academy of 
Sciences. Included in this plan are specific recommendations on service delivery and system 
of care research as well as suggestions on ways to eliminate impediments to research such as 
the lack of human and financial resources needed to adequately stimulate the field* 

On May 1 - 2, 1990, CASSP sponsored a technical assistance meeting devoted to research 
issues which was held in Washington, D.C. The purpose of the CASSP Technical Assistance 
Research Meeting was to encourage and assist the field, made up largely of university-based 
researchers and public policymakers, in the formation of meaningful, long-term collaborations 
that would ultimately increase their chances of successfully competing for research 
demonstration grant awards (R18s) - that is, research on the efficacy of innovative services 
and systems of care which have been developed largely as a result of CASSP goals. Such 
research requires an understanding of the pertinent theoretical, design, and measurement issues 
that will most likely produce the generalizable knowledge needed to advance the field. 

Although the use of experimental design has been particularly emphasized in these proceedings 
it is certainly not the only means of producing important new scientific information. In fact, 
a number of factors may create the necessity for the use of other designs. One such factor 
is the degree to which a problem or issue has received previous study, forming a foundation 
of knowledge upon which to base hypotheses or research questions. Descriptive research 
approaches may be needed for service strategies or problem areas which have received little 
prior study. Further, many of the technologies for conducting research on service delivery 
and systems of care for children and youth are in need of further refinement. Thus, the need 
for developmental research is also critical. Information about a variety of research designs 
and, perhaps, the need to develop new methods and measures for studying mental health 
services for children, adolescents, and their families will be addressed in greater depth in 
future NIMH activities. 
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The Child and Family Support Branch and its technical assistance and research and training 
centers together with other Division of Applied and Services Research staff are available for 
continuing consultation and assistance in proposal development as well as training and other 
career development needs. 



Diane L. Sondheimer, Ph.D. 
Chief, Research Demonstration Program 
Child and Family Support Branch 
Division of Applied and Services Research 
National Institute of Mental Health 



INTRODUCTION 

Ms. Jean Athey, Projcdt Officer 

C3iiW and AdaJesccut Service System Program 

National Institute of Mental Health 

Ms. Athey indicated that ihc National Institute of Mental Health (NIMH) is concerned about 
insuring that the goals of the Child and Adolescent Service System Program (CASSP) are met 
within a research framework. The CASSP grant announcements offer funds for research 
demonstration projects, and NIMH seeks to assist states in preparing the strongest possible 
applications. Thus, the goal of this meeting was to focus on research design, data analysis, 
instrumentation, and other related issues in order to assist potential applications to respond to 
the CASSP research announcements. 



Ira Lourie, MIX, Chief 

Child and Family Support Branch 

National Institute of Mental Health 



Dr. Lourie indicated that CASSP has been operating for five years. When the program 
started, there was very little state planning and commitment in the area of children's mental 
health services. The premise of CASSP was to provide financial and technical assistance to 
states to assist them in creating improved systems of care for children and their families. He 
reported that some systems have now been developed, in some cases in very innovative and 
creative ways, and that there is now an opportunity to begin to ask some serious questions 
about the efficacy of these systems and about the system development process. In the past, 
we were unable to study systems that did not yet exist; however, at this point in the 
movement there are some systems which can be studied. Dr. Lourie stated that CASSP is now 
moving into a new research phase to take advantage of this opportunity. The goal of NIMH 
is to create a knowledge base from which greater system building efforts can proceed. 

Dr. Lourie described the 1990 CASSP grant announcements and elaborated on several key 
aspects. He defined a research demonstration as taking the state CASSP process and applying 
it to the local level within a research framework. "We have to ask questions and aid our 
process in a way that those questions can be answered in a reasonable way." One of the 1990 
research demonstration priorities is for research on "innovative service system development 
and functioning." This concept rests on the notion that there are systems of care being 
developed through CASSP, the Robert Wood Johnson program, the Casey Foundation initiative, 
and by individual states. These system building efforts a'e well funded, and they should be 
generating questions. The CASSP announcement offers the opportunity to receive research 
funds to address some of these questions. 

A second 1990 priority is comprehensive service research for homeless adolescents who have or 
are at risk for severe emotional disturbance. This provides an opportunity for those working 
with this population to develop systems and to study how the systems work and their effects 
on the health, mental health, and social welfare of homeless adolescents. 

Dr. Lourie closed by noting that this conference was intended to be an opportunity to explore 
the research issues and how to design research projects in response to the CASSP 
announcements. 
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RELATING CASSP GOALS TO A RESEARCH DESIGN 



Robert Friedman, PhD., Director 

Research ind Training Center for Children's Mental Health 

Florida Mental Health Institute 

University of Sooth Florida 

Dr. Friedman provided a context for the CASSP announcements by discussing "where we are in 
the field" and some of the issues that are confronting those who are attempting to improve 
services for children and adolescents with serious emotional disorders and their families. He 
reported that a study of key informants was completed several months ago in order to obtain 
their perspectives, as CASSP turned five years old, on what has been accomplished. There 
appeared to be widespread consensus that, in its five years, CASSP has had enormous success 
in a number of areas, including: 

o Increasing the attention and recognition of the needs of children and adolescents across 
the country. 

o Strengthening leadership at the state level in the children's mental health area. 

o Creating a network of information exchange and support amongst people in different 
states. 

o Creating a vision of what a system of care should be, based on the best available research 
data and emphasizing the need for systems to be community-based and comprehensive, to 
use individualized approaches to treatment, to have a strong family focus and to involve 
families in all aspects of service delivery, and to be sensitive to the needs of culturally 
diverse groups and to be culturally competent. 

o Generating significant new resources for children's mental health in many states. 

Dr. Friedman also noted that legislation is pending at the Federal level which would provide 
funds to help states build and strengthen their systems of care. He emphasized that it does 
not do a lot of good to study systems unless you have systems at a state of operation that 
can be effective. The Children's Community Mental Health Services improvement Act of 1990, 
if funded, weald be a very significant effort in this direction. 

Despite a lot of progress, Dr. Friedman acknowledged that there is a long way to go in 
operationalizing our vision and values. "And there are large discrepancies between what the 
state of knowledge is and what we are actually doing out there, the numbers of children and 
families that need help and the numbers who are actually receiving help, and what we 
emphasize in terms of family involvement and cultural competence in relation to what is 
actually being done around the country. We have a long way to go . . . And if we are to 
continue to move the agenda that we have for children and adolescents ahead, it is important 
that we increase the base of support for our goals, the CASSP goals, for children and 
families.* Dr. Friedman stated that in order to move this agenda ahead, it will be necessary 
to tackle more difficult and complex issues, to put our assumptions and beliefs to stringent 
tests, and to sharpen our thinking about the linkages between our activities and the outcomes 
that we seek to achieve as well as to consider other approaches to achieving these outcomes. 
Further, we need to gather more compelling data and research evidence with which we can 
reach out to policymakers and to our own professional colleagues so that we don't remain *a 
small dub,* one with strongly held beliefs but which remains a very silent part of the field. 
In addition, we need to be continually open to learning better ways of doing things. In this 
context, the CASSP research demonstration announcements provide an exciting opportunity. 
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Dr. Friedman also recognized that increased research is needed as more and more states and 
communities become involved in system improvement efforts. Tm sure you've had this 
experience in your state. The question is asked, 'Do we know enough about what works to 
justify supporting more services in the communities? Is there a sufficient empirical base? Is 
there a sufficient research base for us to go ahead and invest significant resources in systems 
of care?* And I think that, as money is tighter and tighter, those questions are going to be 
asked more and more, and it's going to become harder to convince people unless we get the 
kind of research evidence that I think we all acknowledge is needed." 

Dr. Friedman identified a series of critical needs for research. He first suggested that the 
interventions implemented and studied must be meaningful and important in terms of CASSP 
goals and system outcomes. Potential research issues include*. 

o Research on Alternatives to Residential Treatment - This may involve exploring 
alternatives to out-of-state placement or out-of-county placement. In many states, at 
least two-thirds of the budgets are going to out-of-home placements, and there is no 
research base to support the efficacy of these types of services. As long as systems 
continue to be out of balance, the resources are not likely to be available to develop 
community-based services. States have implemented a number of strategies to reduce out- 
of-home placement including: providing new services, developing individualized treatment 
approaches, changing the structure of the service system, cha n gi n g the responsibilities of 
different agencies, changing the responsibilities of the local level in relation to the state 
level, changing reimbursement systems, creating more fl-xible pots of money to allow 
treatment to be more individualized, creating new structures that change the decision- 
making or the gatekeeping process by which youngsters are removed from their homes and 
placed in out-of-home placements, improving the data collection system within a state, and 
providing more accountability at the local level. Training is another strategy that states 
have used in an attempt to reduce out-of-home placements. Some have focused their 
efforts specifically on youngsters who already are out-of-home or who are at high risk of 
out-of-home placement. In this area, Dr. Friedman stressed that it is important to look 
not only at system outcomes (such as broau numbers of youngsters who are placed and 
length of stay) but also to combine this with individual child and family outcomes 
including clinical measures and measures of functioning. 

o Research on Innovative Services - Many innovative services have developed, particularly in 
the 1980s. These include individualized treatment approaches, case management with 
wraparound services, mobile crisis services, intensive home-based services, respite care, 
parent support, therapeutic foster care, and day treatment. While our focus on 
community-based systems is essential, we also need to remember that some of the 
components of the system of care have not yet been adequately studied, and it is 
reasonable to develop research demonstrations that look at some of the critical 
components within the system. 

o Research on Reimbursement and Funding Strategies - This area involves creating and 
studying fiscal incentives for home and community-based services. Some of the Medicaid 
changes offer compelling opportunities for states, the new EPSDT regulations, for example. 
Many states are moving toward creating flexible pots of money, but there has been no 
research on the effectiveness of flexible funding. Suppou for this concept could be 
generated much more rapidly if there were good research. Additionally, though it may be 
beyond us, research is needed on different reimbursement mechanisms. 

o Research on Involving and Supporting Families - This has been a critical focus of CASSP 
since its earliest phase and there are many related research opportunities. For example, 
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people consistently recognize the need' to involve families in treatment planning, but there 
are few people who feel comfortable about the strategies they have developed to achieve 
this, and there has been little evaluation. Research is needed to compare different 
strategies or to evaluate strategies from the family perspective, as well as from the 
treatment plan perspective, to determine the outcomes of involving parents in different 
ways. We invest considerable time and money in training parents and professionals to 
understand each other's roles and to produce closer partnerships between them, and there 
are many research possibilities around such framing as well as around parent support 
groups and different strategies for developing them. It also is important to look at the 
relationship of involvement of parents to satisfaction, and the relationship of both 
involvement and satisfaction to outcomes for the child and family. 

Research on Multi-Agency Collaborations - This also has been a part of the CASSP 
philosophy from the beginning. One of the things that all CASSP states have done is to 
develop some type of interagency entity, cluster, case review team, or planning structure. 
How do we do this effectively? What are the outcomes? Do they, in fact, produce 
different types of treatment plans? Do they, in fact, produce more joint funding or 
sharing of information between agencies? Different kinds of services? Dr. Friedman 
suggested that this is an area in which we have accepted that this is desirable without 
putting it to the test. There are strategies emerging for measurement of 
inter organizational relationships which may be useful in looking at interagency 
communication networks, resource sharing, and other aspects of interagency collaboration. 

Research on Developing Culturally Competent Systems of Care - A key goal of CASSP is 
recognizing the increasing cultural diversity of our country and ensuring that services are 
responsive to the needs of individuals of varying cultural and ethnic backgrounds. 
Research is needed to look at the efficacy of different service approaches for different 
groups, approaches to identifying those in need within different groups, the effectiveness 
of different strategies for changing professional actions and attitudes, and the 
effectiveness of changes in policies. Studies are needed to determine how we go about 
operationalizing and implementing the value of cultural competence and determining which 
strategies achieve the best outcomes. 

Research on Assessment of Treatment Planning »* d Clinical Decision Making - The heart 
of what we are trying to do with children and families is to develop treatment plans that 
are individualized. Yet, we have done very little research within our systems on the 
assessment and treatment planning process. What are the effects of different approaches 
to information gathering? For example, what are the effects of approaches that are more 
ecological in focus and approaches that are more focused on the strengths of youngsters 
and families and on their cultural heritage? Do these approaches result in different 
treatment plans? In better treatment plans? In better outcomes? In addition, little 
research has been done on the group treatment planning process - the organization of 
groups, the structure, membership, and process by which they operate. It may be useful 
to look at the mechanisms we use for developing treatment plans and whether they, in 
fact, result in more creative processes. 

Research on Cost of Care - This area includes the issue of cost of care and cost of 
noncaie as well & who bears the cost (public system, private system, or parents). It is 
going to be necessary to do a far better job of clearly documenting the cost of care. 
Two of the projects which have had the greatest impact in the country are the Ventura 
County Children's Mental Health Demonstration Project and the Alaska Youth Initiative. 
If you ask the people who designed and managed those projects, they will tell you that 
the impact of their projects can be attributed in part to the measurement of outcomes and 
the systematic gathering of data, including data on cost. 



In closing, Dr. Friedman reemphasized the importance of focusing on interventions that are 
meaningful to the system in terms of the goals of CASSP and of the state and of relating the 
intervention to those goals conceptually. Interventions must be powerful enough to have some 
effect in achieving the goals and must be embedded in a design that will yield a dear 
interpretation of the findings so that we can have some generalizable conclusions that will 
allow the entire children's mental health field to move ahead. For additional discussion of 
research needs, see "Research on Service Delivery and Systems of Care: Recommeniaions for 
the NIMH Child Mental Health Research Plan," included as Appendix A. 

Len Bickman, PhD., Director 
Center for Mental Health Policy 
Vandcrb&Unfversity 

Dr Bickman discussed the grant review process and some of the problems identified in 
previous applications. He first described the Initial Review Group (IRQ) ^at is organized by 
an Executive Secretary who is independent of the CASSP program office. This reflects a 
strong attempt by the government to separate funding decisions from the program offices. 
Program staff are not permitted to address the review group unless the Executive Secretary 
allows this. Currently, there is no standing review committee in the children's area, and so 
each year the group is reconstituted. While some people may serve again, others may not, 
making it hard to predict the culture that may develop during the review, the criteria the 
committee will think are important, or the projects they think should be funded. A primary 
reviewer is assigned to each proposal, and this person has a great deal of influence on how 
the proposal will be evaluated and voted on. The entire committee discusses the proposal, 
with some emphasis on the strong points but primarily focusing on the weak points. The 
committee makes a decision about whether or not to approve the proposal. For those that 
are approved, each committee member assigns a priority score ranging from 100 to 500. An 
overall priority score is derived the lower the number, the better the score. A percentile 
score also is attached to each proposal that reflects the ranking of priority scores of all 
proposals reviewed by the committee. 

Dr Bickman then explained the criteria by which a proposal is likely to be assessed (See 
Demonstration Proposal Assessment on page 6). First, he discussed assessment of the 
intervention. Reviewers will attempt to evaluate the quality of the intervention, referred to 
as "program theory." What is the theory behind this intervention? Does it make sense? The 
program theory refers to the linking of the intervention and what it is intended to accomplish 
(its outcomes) and whether the connection makes sense either based upon previous research or 
based upon logic. Dr. Bickman noted that one of the problems identified in previous proposals 
was that the intervention was presented in the proposal with no justification. "You have to 
be able to put the intervention into the context that makes it compelling as to why this 
intervention should be tested. One of the major departures you have to consider is that the 
review committee really doesn't care about whether these children m your state are getting 
these services or not. This grant announcement is not to fund services for children. It is to 
create knowledge about services for children." 

The key according to Dr. Bickman, is to create generalizable knowledge from your 
intervention. Those in other states must be able to replicate what is done. "If its an 
intervention that is very idiosyncratic to one situation, my sense is that's not going to be a 
very appealing intervention. It has to be something that other people have the opportunity to 
apply, because the purpose of the NIMH in the 1990s is to create knowledge, not to provide 
funds to states to provide services." 
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DEMONSTRATION PROPOSAL ASSESSMENT 



INTERVENTION EVALUATION 

A. PROGRAM THEORY A. DESIGN AND MEASUREMENT 

L Problem 1. Internal Validity 

2. Intervention 2. Statistical Validity 

3. External Validity 

4. Construct Validity 



B. IMPLEMENTATION - FEASIBILITY 

1. Organizational 

2. Financial 

3. Human 



B. IMPLEMENTATION - FEASIBILITY 

1. Organizational 

2. Financial 

3. Human 
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Thus, the intervention has to be conceptually sound; the intervention must be related to the 
outcomes that are supposed to occur. Further, the intervention must be policy relevant, and 
reviewers will be looking for interventions that are likely to create large impacts. Basic 
researchers may be interested in very small effects that have little or no policy relevance, but 
prodi " meaningful theoretical gain. For research demonstrations, however, interventions with 
very amall effects are not going to be as credible as something that can really change the 
nature and quality of services and outcomes for children. 

The nature of the problem also must be discussed from a research perspective. Reviewers 
want to see an understanding of the context in which the demonstration will be applied. 
Previous data about the problem in the community or state gives an advantage in the proposal 
as well as a sense that prior work has been done. The demonstration aspect is very critical. 
It has to be relevant and you have to show it's relevancy to solving a problem that you have. 
Which means you need to define the problem. Which means probably, any data that you have 
about the problem, is going to be advantageous in your proposal. Proposals that [only] seek 
to collect date to define the problem will not be very successful." 

In sum, reviewers will look at the quality of the intervention - Is the idea compelling? Is it 
important? Would it have an impact? Is it theoretically meaningful? Is it policy relevant? 
Is it exciting or innovative? 

Committee members also will judge the applicant's ability to implement the intervention. 
States can hire the best proposal writers, but the capabilities of the organization to implement 
the intervention may be questionable. Proposals must demonstrate the organizational capability 
to put the intervention in place including needed connections, power, expertise, and resources. 
"What is it in this proposal that convinces me that this is a high probability event? Because 
I don't want them to spend the research dollars and the demonstration dollars on trying to 
demonstrate the futility of intervening in a system." There must be sense that change is 
possible and that the organization is capable of doing it. The proposal must contain sufficient 
funds to implement the intervention and must clearly indicate where additional funds needed 
to support die intervention will come from. Human resources also are critical. Does the 
director of the project have the experience and capability to run complex demonstration 
projects? The background of proposed personnel will demonstrate whether they have the 
needed experience and capability. 

Dr. Bickman proceeded to discuss criteria for assessing the evaluation aspect of the proposal 
beginning with design and measurement issues: 

o Internal Validity - Did the program cause the change? According to Dr. Bickman, some 
sort of comparison is going to be necessary to demonstrate that, in fact, the intervention 
was the one that did it. 

o Statistical Validity - The design must be sufficiently sensitive to be able to detect an 
effect if it is present. Tins was the major downfall of previously submitted proposals- 
the sample sizes and the implementation of designs were too insensitive to detect effects. 
A reviewer ctrx look at the design, the sample size, the type of instrumentation, and the 
type of analysis proposed and have a pretty good guess that even if the program is 
effective, the evaluation would not be sensitive enough to show that it was statistically 
significant. "All that this does is communicate to the larger field that these child 
interventions don't work, when in fact it was the evaluation that didn't work. And it's a 
critical element to be able to handle questions of statistical power and conclusion validity 
in your design." 



7 

14 



o External Validity * This is a question of generalizability. If there is a very idiosyncratic 
program or very idiosyncratic population, the proposal probably has less of chance of 
being funded. 

o Construct Validity - One aspect of construct validity relates to measurement or 
instrumentation. Does it measure what you say it is measuring; Is it valid and reliable? 
The construct also refers to the program itself. It is important to describe the 
intervention not only in operational terms but to raise it to a generalizable construct. 
"So, what do wraparound services do that should make it better for the child and family. 
If you only describe operationally what you're going to do without describing why it 
should be better, it's going to be less conceptual." 

Dr. Bickman emphasized the importance of a good evaluation component in the proposal. 
"Essentially what the review team is doing is differentiating proposals how they differ, 
which ones are fundable, which ones are better than others ... My sense is that the criteria 
for what is a good research design are clearer than for what is a good intervention. And the 
ability of a member of the committee to communicate those criteria to the rest of the 
committee is easier for research and evaluation criteria than it is for interventions ... So 
discussion then falls more about what the evaluation looks like." Measurement issues, 
therefore, are very important. 

Decisions will have to be made about how much money to put into the evaluation versus how 
much money is put into the intervention, and balance is an important concept. "Someone may 
say they want to provide this important intervention and it costs a lot of money to put it in, 
and they have to decide between having a control group or having a stronger intervention. 
These things, remember, go back to the basic purpose of these grants, which is not to provide 
sendees to children in your state, but to provide generalizable knowledge about what happens 
when you provide these services ... So in terms of dollars, you have to be careful about how 
much money you put into services versus evaluation." Additionally, there is a sense that state 
generally can find service money somewhere else but cannot find research or evaluation 
dollars. This decision requires some hard choices; Dr. Bickman stated that his bias is to have 
the best evaluation design. 

Dr. Bickman summarized important criteria for both the intervention and evaluation aspects of 
the project. The section of the proposal describing the intervention should demonstrate: 

o Balance with the evaluation in terms of attention in the proposal and resources. 

o Data-driven local description of the problem. 

o Clear and conceptual description of the intervention. 

o Ability to coordinate and manage the intervention. 

o Sufficient resources for the intervention. 

o High quality and experienced staff. 



The component addressing the evaluation requires: 

o Comparison groups (internal validity). 

o Power analysis/statistical plan (statistical validity). 



o Precise description of instruments (construct validity), 
o Description of sampling of sites and clients (external validity), 
o Ability to control design and sample (organizational resources), 
o Sufficient funds for the evaluation (financial resources), 
o High quality and experienced staff (human resources). 

The proposal must show how random assignment will Ik done if that is proposed, where 
comparison groups will come from and that there will be cooperation from people who will be 
in comparison groups, evidence that you have access to the number of clients promised in the 
proposal, and so forth. 

Proposed staff should include a team that is capable of implementing a complex field study 
which involves gaining access to clients, collecting data, and analyzing data, all within a short 
period of time. The best way to demonstrate this clearly is that proposed personnel have a 
track record in terms of successfully submitting other grant proposals, conducting studies, and 
publishing. 
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ISSUES IN RESEARCH DESIGN 



Michael Hendricks, PIlD., Consultant 
MH Associates 

Dr. Hendricks discussed the topic of research design. He reviewed six important questions 
related to research design (See Issues in Research Design," Appendix B). The first three 
questions relate to the demonstration or intervention (What do I want to change?, How can I 
measure these changes?, and How can I intervene to produce changes?). The remaining three 
questions relate specifically to the research design (How can I detect any changes which do 
occur?, How can I be certain that my intervention caused these changes?, and How can I 
ensure that others can learn from my success?). 

t What do I want to change? 

The first decision that this question requires is what the level of focus mil be. Are you 
going to try to change things at the state level or at the local system level? Are you going 
to try to change service delivery or are you going to try to change client outcomes? This is 
an important decision which leads to identification of the "units of analysis/ This simply 
means what it is that you are trying to affect. If you are trying to affect systems, then 
"systems" is what you want to analyze. You may be trying to affect families or children. 
The level of focus and units of analysis are two issues that are very much interrelated. 

This question also requires ;i delineation of the specific problem to be addressed, the 
magnitude of the problem, and trends related to the problem. This is the "data driven local 
description" which makes the case that the project addresses a difficult issue. 

2. How can I measure these changes? 

This question addresses the evidence that you will be providing to show that you have made 
changes. Relevant issues for this question are validity, reliability, and sensitivity to change. 
Validity is simply that the measures you are using in fact measure what they are intended to 
measure. There are four ways to demonstrate that your measures are valid: they are 
consistent with common usage; they are consistent with alternative measures that others have 
used; they are internally consistent; and they predict future behavior. 

Reliability also must be demonstrated, meaning that if you measured again next month, if 
there had been no change, you would get the same results. Composite measures almost always 
are better than separate measures as they are less unstable, and multiple measures are better 
than single measures. Additionally, uniform data collection techniques and instruments are 
necessary. 

Sensitivity to change is another important measurement issue. You must have some room on 
your measures in order to have the opportunity for things to move and to show an effect. 
Having a range of possible responses is one approach, and continuous measures are better than 
categorical measures. In addition, it is necessary to guard against floor and ceiling effects, 
ensuring that the people you sample or the measures you use are not bumping up against the 
ends of scales so that there is no room to move and show a difference. 

3. How can I intervene to produce changes? 

This question addresses the intervention needed to produce changes and also to prove to 
others that you have made some changes. First, there must be a good concept in the 
intervention, a theoretical base. What exactly about your intervention is supposed to be 
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causing the effect? Several references listed in Appendix B, particularly Program Theory in 
Evaluation, may be helpful in this regard. 



Regarding implementation of the intervention, it must be feasible. You must have the 
experience and the resources to do it, and not only that you can do it, but that other people 
can do it too if it turns out to be a good idea. The intervention must be monitored to ensure 
that it's being implemented, and proposals should indicate that you will monitor the 
intervention and mil know if it is not being implemented well. Further, the intervention must 
be well documented so that if it does work, someone else will know what you did. 

Finally, you want to have little "diffusion of treatment" or "leakage" or "contamination" 
between treatment and control groups or these groups will begin to look alike. 

I. How <r "l I detect any changes which do occur? 

It is necessary not only to detect any changes which occur, but to demonstrate these changes 
to other people. This apparently was a major downfall of previous proposals. People were 
not able to show in their proposals that if there was an effect, they would able to show it in 
fact. The requirement here is to have the strongest statistical power you can possibly have. 
There are four considerations to maximize your statistical power: 

o Increase sample size - Virile this is the first approach we think of, it is not always easy 
or even possible. There may only be so many counties in your state, and it may be too 
time consuming or expensive to add more families or children to your sample. While you 
may try to increase your sample size, there are other solutions as well. The book entitled 
How Many Subjects? Statistical Power Analysis in Research may be a helpful reference. 

o Increase effect size - A second way that you can increase your statistical power is to have 
a larger effect or to be able to show the effect better. Strategies include ensuring that 
the intervention is strong and powerful, ensuring the integrity of the treatment, and 
ensuring that data collection is consistent and sensitive to maximum impact. Maximum 
impact means that if you have worked with families and you think that they are going to 
benefit most two months after you have worked with them, then measure them two months 
after you have worked with them. Don't measure them eight months after. Give yourself 
the best chance to show any effect that is there. 

o Tests conducted - A third way you can increase your statistical power is by the kinds of 
statistical tests that you conduct since some are more powerful than others. For instance, 
parametric tests are almost always better than nonparametric tests. T-tests, analysis of 
variance, and multiple regression are almost always better than tests like chi squares, even 
though chi squares are what most of us do all the time. There are many better variations 
on chi squares. "Another consideration is that of reducing extraneous variation. In other 
words you want to clear away a lot of the underbrush, so if you've grown a flower you 
can actually see the flower. That's the way I think of this/ One way of reducing 
extraneous variation is to learn as much as possible about factors which might influence 
peoples' progress such as gender, age, types of problems they are exhibiting, etc. These 
factors can be used as covariants, control variables in regression equations. This goes 
back to having a stratified sample. 

o Alpha level • The fourth way to increase your statistical power is to change the alpha 
level, or the legendary p < .05. There is no magic to this confidence level or alpha level 
which specifies that 95 out of 100 times there's an effect and that 5 percent of the time 
you're going to be wrong. In public policy research, it would be acceptable if you were 
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right only 90 out of a 100 times or perhaps even less. You may be able to make a good 
case for changing the confidence or alpha level to .10 or .20. 

5. How can I be certain that my intervention caused these changes? 

The following graphic displays a common pretest-posttest design. It suggests that behavior 
improved dramatically from pretest to posttest and that the intervention was highly effective. 
However, the improvement may not have had anything to do with the intervention. 



THEORETICAL RESULTS: 

PRETEST-POSTTEST RESEARCH DESIGN 



IMPftOVID MM* Wft 




TIME 

■i cMLOfwra MH/moR 



Other possible explanations include: 

o History - Other things may have been going on in between. 

o Maturation - The children may simply have gotten older and settled down. 

o Testing - Based on familiarity and experience with the test, the children may have 
performed better on the test the second time. 

o Instrumentation - There may have been a change in measuring instrument or perhaps 
caseworker observations were more sympathetic and positive the second time. 

o Regression to the Mean - This is a concept that if you select the most difficult cases to 
work with, even if the intervention is completely ineffective, the children will almost 
certainly look better on the posttest. The explanation is that you measured them 
specifically because they were very extreme scores, and statistically they almost have to 
come back towards more of an average score the second time around since it would be 
difficult for them to get worse. 

o Mortality • There may be children who drop out of the study, and it's possible that the 
most difficult youngsters dropped out leaving better children for the posttest. 



It is critical to reduce threats to validity and to show k your proposals that you have 
reduced threats to validity to the lowest possible level. Pre-experimental designs (performance 
monitoring, pretest-posttcst (one group), case studies, and cross-sectional surveys are the least 
desirable designs. Quasi-experimental designs are better designs and include non-equivalent 
control groups and longitudinal time series. With non-equivalent control group designs it is 
important to make the groups as equal as possible and to do something with one group and 
not with the other. The use of matching in the different groups can increase rigor. In the 
longitudinal time series designs, it is important to have a lot of data points, at least 30. 

If it is possible to move up to a randomized experiment, this is the best design of all and is 
not as difficult to do as most people think. Pretest-posttest with two or more groups is a 
very common design, but a design that is equally as rigorous is a posttest only as long as 
there are two or more groups in random assignment. 

6. How can I ensure that others learn from my success? 

Can the results be generalized? Two aspects of this involve determining: 1) if results can be 
generalized beyond the study sample to your entire target population, and 2) if results can be 
generalized to other populations or to other states. This issue is addressed in the sampling 
plan. In the sampling plan, the populations you will be dealing with should be identified very 
clearly. Stratifying the sample in terms of children with very difficult problems or certain 
kinds of problems is beneficial, and random sampling is most desirable in terms of enhancing 
generalizability. 



John Burchard, PhJX, Professor 
Department of Psychology 
University of Vermont 

Dr. Burchard described a particular kind of design called the time series analysis or repeated 
measures design. This design is being used in Vermont to evaluate services for severely 
emotionally disturbed children and adolescents. Dr. Burchard noted that his research 
environment is the State of Vermont and that an important part of that environment is the 
Vermont State Legislature. A major problem to legislators in Vermont and elsewhere is that 
we do not have outcome data on the children we serve. When Dr. Burchard was Commissioner 
of the State Child Protection Agency, legislators asked him many pointed questions: "Where 
are the severely emotionally disturbed children you served five years ago? How many have 
graduated from high school? How many have a job? How many are in the correctional 
system or the state mental hospital? You don't know! Why don't you know? I know where 
my cows are. I can tell you whether or not the services for my tractor are any good! Why 
can't you tell whether or not your services for kids are any good?" Dr. Burchard explained 
that when he left state government and went back to the university, he sought to build a 
collaborative process between the state and the university to determine what happens to the 
children we serve and what works best for what kind of children. 

There is very little data available in Vermont (and in any state) on either clinical outcomes or 
on the level of restrictiveness of the environments in which children are served. The 
approach described by Dr. Burchard is in the developmental stages and is designed to collect 
data to determine if children are improving and if they are moving to more normalized 
environments. 

Currently, there are 20 children on a system which tracks them on a daily basis. This 
information is converted graphically into weekly or monthly data. The categories of variables 
that are tracked include the adjustment of the child and the level of restrictiveness of the 
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child's environment. Measures that will monitor the services received and the satisfaction of 
the youth and parent will be implemented in the near future. For daily adjustment, a series 
of indicators of both positive and negative behaviors are recorded. These indicators include 
the following: 



0 


Physical aggression 


0 


Sexual abuse/assault 


0 


Property damage 


0 


Suicide attempt 


0 


Theft 


0 


Fire setting 


0 


Runaway 


0 


Strange behavior 


0 


Alcohol/drug abuse 


0 


Cruelty to animals 


0 


Sexual acting out 


0 


Police contact 


0 


Extreme verbal abuse 


0 


Self-confidence 


0 


Sad 


0 


Compliance 


0 


Anxious 


0 


Peer interactions 


0 


Self-injury 


0 


School attendance 


0 


Inappropriate bowel movements 


0 


Vocational involvement 


0 


Life threat 







These 23 indicators comprise the Daily Adjustment Indicator Checklist shown on page 15 and 
16. An assessment is made as to whether these primary indicators are changing over time. 
However, it is also important to look at the child's environment since an equally important 
goal is to move the child to the most normalized environment possible. Thus, in addition to 
adjustment indicators, the project is involved in tracking children along the following 
continuum of residential restrictiveness: 



o Independent living 
o Home 

o Relative/mentor 

o Foster home 

o Group home 

o Residential treatment center 

o Psychiatric hospital/corrections facility 

The goal is to make the data collection system "provider friendly by limiting the data 
collection task to five minutes per day and by providing periodic graphic feedback to the 
caretaker and the case manager. From a case management standpoint, the data can be 
analyzed on an individual basis, and each child can be used as his or her own control. THis 
approach also lends itself to a group design in which groups of youngsters receiving two 
different kinds of treatment might be tracked. For example, you could have two groups of 25 
youngsters receiving two different types of care (e.g., therapeutic foster care versus group 
home care). The design would be more powerful if the children were randomly assigned to 
the two groups. Samples of graphs resulting from both individual and group designs are 
shown on pages 17 and 18. 

Dr. Burchard emphasized that severely emotionally disturbed children receive many different 
placement changes and that many different services can be included in an intervention 
package. Therefore, he prefers going beyond the traditional pretest, posttest and follow-up 
data points and obtaining more frequent measures of adjustment. This approach is ongoing, 
proactive, looks at rates of change, and provides continuous follow-up over time. In a sense, 
the evaluation never stops. From the perspective of legislators and policymakers, more 
frequent outcome data is more user friendly. However, the approach is labor intensive and, 
therefore, may be more costly. The advantages and disadvantages of the approach are 
summarized on page 19. 
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DAILY ADJUSTMENT INDICATORS 

Directions: Please indicate according to your best judgement whether or not the 

following behaviors or events have occurred on this day. Fill in the appropriate circle. (Y=YES, N=NO) 



ID*. 



Week beginning 



(MONTH/DAY/YEAR) 



1. PHYSICAL AGGRESSION: Did the child or youth hit strike, bite, or scratch a person 

with intent to harm them? (Includes hitting with an object.) 

2. PROPERTY DAMAGE: Did the child or youth damage property on purpose? 

3. THEFT: Did the child or youth take property without permission? 

4. RUNAWAY: Did the child or youth run away? 

5. ALCOHOL/DRUG USE: Did the child or youth use drugs or alcohol without permission? 

6. SEXUAL ACTING OUT: Did the child or youth engage in inappropriate sexual behavior 

which was displayed publicly or directed toward another person? 

7. EXTREME VERBAL ABUSE: Did the child or youth speak to another person in an extremely 

malicious, abusive or intimidating manner? 

8. SAD: Was the child or youth sad, withdrawn, or depressed to a degree which significantly 

interfered with participation in an important activity? 

9. ANXIOUS: Was the child or youth feaiful, anxious, or worried to a degree which significantly 

interfered with partidpation an impo^ activity? 

10. SELF-INJURY: Did the child or youth harm or attempt to harm him or herself nonacciden tally? 

th INAPPROPRIATE BOWEL MOVEMENTS: Did the child or youth intentionally smear 
his or her feces or deposit tbem in an inappropriate place? 

11 LIFE THREAT: Did the child or youth threaten ox engage in physical assault in a manner which 
which you believe was life threatening? 

13. SEXUAL ABUSE/ASSAULT: Did the child or youth attempt to force him or herself upon 
another person sexually? : 
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DAILY ADJUSTMENT INDICATORS, SIDE 2 



14. SUICIDE ATTEMPT: Did the child or youth attempt to commit suicide? 

Si 



M T W T F S S 

©e ®6 ©e ©8 ©e ©e ©e 



^^ : ^e ; c1ttia^6^'ie^ 



in a manner which could have resulted in property damage or harm to others? I-:?.'-. ©e ©e ©© ©8 99 89 ©9 

16. STRANGE BEHAVIOR: Did the youth have delusions, hallucinations, obsessions, compulsions, M T W T F S S 

" 98 89 ©9 ©e 99 ©e 99 

M T • W v ' t : ''-'f' *•% • • S '" 

99 99 99 ©9 ©9 99 99 



v.:-va:>x->:<:.: 



or other bizarre behavior which significantly interfered with an important activity? 
17. CRUELTY TO ANIMALS: Did the child or 



:• v;<-: :•;•:•;•:<•;•:•.• >.-y. : w.-. .•.vx : ;v,*-: . 



18. SELF-CONFIDENCE: Did the child or youth appear self-confident in Ms or her activities for 

more than 85% of the time? 

19. COMPLIANCE: Was the child's or youth's response to requests and general activity 

20. PEER INTERACTIONS: Did the child or youth have good peer/sibling relations 

85% of the time? (Please check □ if no peer or sibling contact on this day.) 

21. SCHOOL ATTENDANCE* As 1st as you know, did the Child or youth receive credit 
- lor school attendance? (Please check Q if there was no school today. ) - 



M T W T F S S 
99 99 ©e ©9 ©9 ©O 99 

99 99 ©e 99 ©e ©• 

M T W T F S S 
99 ©9 ©9 99 ©9 99 99 
□ □□□□□□ 

r* ^^m^mt:" f^-«^ : s :;: 

98 99 99 98 99 99 99 

t' : .o. ?> □ as □ □ □ ■ 



M T W T F S S 
99 99 99 99 99 99 99 



21 PARENT CONTACT: Did the child or youth have contact with his or her natural or adoptive 
parentis) on this day? (Includes letter, telephone call, or personal visit) 

23. VOCATIONAL INVOLVEMENT: Did the child or youth work for pay (check "P") or as X^J^^l V a * 

24. POLICE CONTACT: Did the child or youth have contact with the police concerning M T W T F S S 

his or her negative or suspicious behavior? 99 99 99 99 99 99 99 



25. POSITIVE: What did the child or youth do today that was eood? 



<m 



4^ 

ijiftiihifr^jii-my 
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GENERAL COMMENTS:. 
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INDIVIDUAL DESIGN 




10 
9 
8 
7 
6 

RESTRICTIVENESS 5 

4 
3 
2 
1 
0 
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- PSYCH / CORRECTIONAL 



- RESIDENTIAL 



- GROUP HOME 



- FOSTER HOME 

- RELATIVE / MENTOR 

- HOME / INDEPENDENT 



i 1 1 1 1 1 1 1 1 r 

1 2 3 4 5 6 7 8 9 10 11 12 

MONTHS 
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TIME SERIES 
(COMPARING TRENDS) 



ADVANTAGES 

1. Data Quantity/Quality 

more data 
better reliability 
better internal validity 

2. Ongoing Vs. Time Limited 

proactive 
rates of change 
continuous follow-up 

3. User Friendly 

caseworker/provider 

legislator 

policy maker 



Disadvantages 

labor 

cost 

norms 
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Currently, much of the data is collected by undergraduate students involved in a two-semester 
commitment for course credit. The students train the providers to complete the daily forms, 
enter the data into the computers, and display it in graphic form. A seminar that goes along 
with the data collection efforts addresses issues in the system of care for se^rely emotionally 
disturbed children. 

Dr. Burchard noted that one of the challenges in the evaluation of our service systems is that 
services are very "component" oriented. A child frequently goes from one service component 
to another. Therefore, it is desirable to have a system which tracks youngsters across time 
and across service components. In order to provide individualized care, the money has to 
follow the child. Similarly, this is a data collection system which also follows the child. 
Additional information about the Vermont tracking system is provided in Appendix C. 




SAMPLING ISSUES 

Philip hut, PhJX, Associate Professor 
Center for Health Policy and Research 
Yale University 

Dr. Leaf discussed the importance of the sampling process which involves identifying and 
allocating potential subjects for the study. He noted that providers may become upset at the 
prospect of randomly assigning only some of their clients to what is likely to be improved 
treatment. He suggested framing the issue of randomizing in terms of scarce resources. 
Given scarce resources, we are working in environments in which we are not going to be able 
to provide all the services to all children in need. Therefore, randomizing can potentially be 
seen both as a fair and equitable way of allocating scarce resources as well as a way of 
learning and drawing inferences from a study over time. Presumably, if it can be 
demonstrated that the services work, more resources would become available in the future, and 
additional children would be able to benefit. "One of the things that needs to be discussed in 
a very straightforward manner with providers, clinicians, and planners is the reality of the 
situation, that we are not talking about the opportunity to get as much money as we possibly 
want where everybody could get all possible services. There are limitations. And one of the 
ways of allocating limited resources in a fair and equitable way is to have some people go 
into one program and some people go into another." 

Dr. Leaf suggested that rather than assigning groups to standard treatment versus a new 
service approach, studies might consider allocating clients to two competing programs. This 
approach may be more attractive to providers and families as well as review panels. The two 
programs may be at different levels of intensity or have different components, and it would be 
important to learn about the relative benefits of the two programs. 

One consideration in the research demonstration project is where randomization is 
implemented. There is not necessarily a reason to randomize clients before they have any 
contact with the agency or even before they have a prolonged contact with the agency. With 
regard to the CASSP model, we are interested in how a network of services works. It would 
be legitimate to look at clients after they have an intake, or after they receive a basic set of 
assessments, or even after some treatment. At any point in the service delivery process, 
there may be alternative approaches or services for children, and research could explore how 
best to utilize these alternatives. There may be points within current service delivery systems 
where these decisions are already being made and where it would be very important to 
determine whether a child should receive one set of services or another, or now intensive 
these services should be, or of what duration services should be. These are all potential 
points for randomization. Dr. Leaf emphasized that giving children worse possible outcomes as 
a result of randomization is not acceptable to either providers or researchers. However, there 
are a lot of points at which randomization can occur, and questions can focus on what aspects 
of services work and for what types of children certain services work. 

Dr. Leaf indicated that in planning a research demonstration project, administrative data is 
needed in order to ensure that you can get enough children into the project, so that after the 
three years it will be possible to determine whether or not there was an effect. It is 
important to specify how children will be identified, the characteristics of children to be 
included, and any exclusion criteria. There may be reasons to exclude a small number of 
children with specific characteristics from a broader evaluation that makes the study more 
acceptable or because the intervention or experimental component of the project is not really 
what they need. For example, you would not want to randomly allocate children at immediate 
risk of suicide to a program where someone comes to see them every two weeks. 
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Sample size is another issue to be considered* In order to increase precision, increasing 
sample size may be the easiest thing to do, although it often is very expensive. Some of the 
other things that may be done to improve studies are more difficult, such as increasing the 
precision of measures. Increasing sample size generates cost both in terms of the services 
and the evaluation, especially in terms of the CASSP model which talks about multiple 
services. On the other hand, you do not want to have such a large sample in a research 
demonstration project that you lose control. Too large a sample may make it difficult to 
provide a consistent intervention and to control the quality of the intervention. Thus, there 
is a tension between optimal sample size for a study, how many people can be served, and 
how much money will be available for the project. 

Dr. Leaf noted that there are a lot of questions that we do not need large samples to address, 
many of which are important questions to policy makers. Generally, we are not measuring 
subtle symptomatology or subtle changes in children. Rather, these youngsters are failing in 
school, using illegal substances, getting arrested, being abused, and the like, and it is possible 
with relatively small sample sizes to find effects relative to these types of indicators. The 
types of effects we expect from our programs and the costs of these programs are fairly 
dramatic, and we do not necessarily need large samples. 

Sampling for the evaluation must be integrated into the service delivery program. The service 
program may have eligibility requirements, and therefore some of the intakes in the program 
may be coterminous with the evaluation. There may be information that is obtained in the 
identification and assessment of children that is used to define these children as part of the 
target population. Thus, it is important to integrate the sampling and measurement for the 
evaluation with the intake and assessment process for the clinical aspects of the projects. 
This ensures that the procedures used for the evaluation are as natural as possible. 

The issue of sampling really comes down to the inferences you want to make, according to Dr. 
Leaf. You must determine how many people you will need to have in the demonstration 
project before you are able to draw inferences that are important to you. In research 
demonstration projects, there typically is a package of services or a program that you want to 
implement and answer certain related questions - Did it work? For whom did it work? What 
were the costs? Did the children who received these services do bettor than they would have 
under another legitimate alternative (either the current system if that is seemingly legitimate 
or another reasonable alternative)? The answers to these questions will inform the further 
development of service systems, and the sample must be large enough to address the particular 
questions. 

The methods section of the proposal should indicate how you intend to determine whether the 
services work better for certain groups. Accordingly, the goals and priorities of the project, 
the sampling, and the analysis plan must be. consistent. Dr. Leaf indicated that the fewer 
aims in a proposal, the better, and the sampling plan should be developed to maximize 
information about the question that is of primary concern. He emphasized the complexity of 
our service systems, noting that rather than focusing on the effects of the entire system, it 
may be appropriate to look at the effects of pieces of the system which are varied for some 
of the children. It will not be possible to randomly allocate children to one set of programs 
and then have the schools handle them differently just because they are in your study. 
Generally, schools will not be part of the fundamental manipulation; they are going to be 
handling children over time as will juvenile justice systems and child welfare systems. The 
more complex the question to be studied, the more other parties that may influence the 
selection of a sample. 

It is important to keep in mind that not everybody goes through these programs to 
completion, since people may drop out or move. Thus, when you determine how many people 
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you need initially, you must estimate how many people you will end up with due to attrition. 
The question is not how many people you enroll, but at the points at which you collect data, 
how many people will you have left. Additionally, you must consider how you will follow 
those in the study. While those in the intervention may be easier to track, you must also 
track those in the comparison group who need equal attention in terms of monitoring and 
evaluation, but who do not necessarily have the same relationship with the agencies involved 

In summary, Dr. Leaf advised handling sampling in a way that there is a comparison between 
alternatives that is meaningful to the agency and meaningful to the child and [arents. This 
will increase the chances of a successful project as well as your ability to convince a review 
committee to approve your project. The more policy relevant the proposal and the more 
consistent the sampling is with the questions being raised, the more likely the proposal will 
emerge with a positive outcome. 
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MEASURES/OUTCOMES - SYSTEM OUTCOMES 



MidiadHeadric^Pia). 
MH Associatei 

Dr. Hendricks reviewed potential measures of system outcomes. These possible measures are 
arrayed according to GASSP objectives, and are included as Appendix D, 



Pha Leaf,PhX\ Associate Professor 
Center for Health Policy and Research 
Yale University 

Dr. Leaf discussed strategies for measuring the relationships and linkages between 
organizations as one approach to the measurement of system improvements. He noted that 
there is considerable experience and expertise in the area of network analysis, although few 
developed methodologies relate directly to children's services. Ongoing work in the area of 
interorganizational relationships is being conducted by: 

o Joe Morrissey at the University of North Carolina who has conducted a number of multi- 
county studies and currently is involved in analyzing the Robert Wood Johnson community 
support projects for mentally ill adults. He has developed a number of instruments for 
assessing interorganizational connections. 

o Oscar Gruske at UCLA's Department of Sociology also has conducted multi-county studies. 

o Pat Dorian at the University of Pittsburgh's Department of Sociology is conducting a 
network analysis in two counties in the Pittsburgh area involving children's agencies, 

o Mary Fennel at Penn State University's Department of Sociology. 

Each of these individuals has developed somewhat different techniques for assessing 
interorganizational linkages and different outcome measures. 

Dr. Leaf emphasized the importance of assessing the relationships among agencies along a 
number of different dimensions as well as to utilize a number of different informants - not 
just organizational participants, but clients, parents, and flinders who will all have different 
assessments of the linkages and how well services are being provided In terms of monitoring 
and understanding the system, it is important to obtain these multiple perspectives. It also is 
important to assess both positive and negative linkages, the effective aspects of linkages as 
well as the hindrances to communication and coordination. 

Another issue to be considered is "organizational set." Not only do individual agencies have 
relationships with each other, but groups of organizations tend to form relationships. The 
pattern of these relationships will influence the ability of the organizations to deliver services 
and potentially can influence the quality of services. Even when there is a single agency with 
fiscal responsibility or overall administrative responsibility for children's services, the provider 
agencies and programs in the community still have a very complex pattern of relationships. 
Further, an organization may have many formal subparts. Analyzing that pattern of 
relationships and services is one of the ways of looking at systems, as well as following 
individual clients through the system. 

Additionally, agencies within a community have relationships with regional and state 
authorities. Within an organizational cluster, there may be agencies with different or 
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conflicting mandates and with different networks of services. For example, the mental health 
system may have one network of services and the education system another. An 
organizational cluster includes all these different networks and how they relate and interact 
to provide services for individual children* 

Some impediments for service delivery arise when the ideology and tasks that are seen as 
important by different sectors come into conflict. The school, for example, may recognize 
that children with severe emotional problems have certain needs, but they may see most of 
those needs being dealt with outside of school due either to the expense of providing services 
or because they legitimately feel that other agencies could provide better services. 

A number of variables relevant to interorganizational relationships are displayed on page 27, 
They include the formality of the interactions, reciprocity, centrality, conflict, coordination, 
density (how many agencies are included), multiplicity (how many different types of services), 
cohesion, etc 

Dr. Leaf noted that, with the exception of Pat Dorian, he is unaware of people who have used 
network approaches to measure the types of exchanges that would be important to CASSP- 
typc systems of care. To some extent, children's service systems are more complex than adult 
systems because of the multitude of agencies and organized sectors of society that are brought 
to bear on providing services to children. However, the involvement of multiple agencies may 
make the approaches of network theorists even more applicable, because they can serve as an 
important data reduction procedure. 

Dr. Leaf described a technique that has been developed to attempt to define what comprises 
the system of services in a community. Because the number of agencies that provide 
important services varies enormously, a snowball technique has been developed. A list of 
agencies or services from any individual typically would be a very limited list. The snowball 
technique involves going to clients, families, provider agencies, planning and administrative 
agencies, funding agencies, etc. to provide lists of agencies and services in the system. This 
technique yields a more complete identification of all involved services. Respondents also can 
be asked to rate their satisfaction with particular services in terms of quality and in terms of 
the positive and negative linkages among agencies. 

The person from whom you obtain information about an organizatui is an important 
consideration. No one person has all the, information, particularly in large or complex 
agencies. Further, if an agency has a number of different subparts that have important roles 
in the delivery of children's services, informants must be identified who can provide 
reasonably good information about each subunit. The director of the agency may not 
necessarily have information on all the subparts and all the relevant linkages and 
interorganizational relationships. 

Dr. Leaf emphasized that there is no one right way of conducting studies examining 
interorganizational relationships, although there are some fairly well developed methodologies. 
He suggested it is important to measure things that are consistent with the priorities of the 
proposal, the way the particular system of care is conceptualized, and the linkages that are 
considered most important. 
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VARIABLES OF IOR STRUCTURE AND PROCESS 



Interaction Formalization 
Interaction Stindtrdiation 
Transaction Importance 
Transaction Frequency 
Reciprocity 
Perceived Power 
Centrality 
Cooperation 

Conflict 
Coordination 



VARIABLES OF NETWORK STRUCTURE 

Density 
Multiplexity 
Cohesion 
Reciprocity 
Structural Equivalence 
Reachability 
Centrality 
Complexity 



Fennell M. & Dockett K. (1990). Interorganizational systems approaches. In J. Morrissey & D. 
Dennis (Eds.) Homeless and mental illness: Toward the next generation of research studies. 
Rockville, MD: National Institute of Mental Health, Office of Programs for the Homeless 
Mentally 111. 
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MEASURES/OUTCOMES - CLIENT OUTCOMES 



Len Bickman, Ph JX, Director 
Center for Mental Health Policy 
Vandcxbitt University 

Dr. Bickman stated that the measurement of client outcomes is the bottom line. "You can do 
all the nice system changes, all the brilliant interorganizational networking, all the provision 
of resources, and do all the things that theoretically or conceptually or as an article of faith 
you believe in. At some point, we are accountable for saying whether the intervention 
affected children, if it affected families. Because if it doesn't, then clearly for the last 
number of years, we have taken a wrong turn. And that, essentially, is the bottom line of 
questioning the assumptions underlying the CASSP philosophy/ CASSP has been based on an 
orienting philosophy about what we need to do to improve the well being of children. 
However, from the perspective of evaluators, these are still assumptions. They have not been 
demonstrated sufficiently, and the purpose of the research demonstration projects is to start 
looking critically at some of these assumptions. The issue is to look carefully at what we are 
doing to make sure that what we are doing actually improves the lives of children and 
families. 

There are many instruments and methodologies for the measurement of clinical outcomes. Dr. 
Bickman advised against designing your own instruments for a research demonstration project. 
A three-year project is difficult at best. If you spend the first year and a half designing 
instruments and pilot testing them, there v ill be little time left for data collection. And you 
cannot go into the field without an instrument that has already been field tested. While you 
may include new instrumentation as ancillary instrumentation, the key instruments you use 
should have a literature published in reputable journals. 

Additionally, use of these instruments requires training to administer them, thought about the 
data analysis process, and a system of quality control for the data collection process. 
Proposals should go beyond describing the instruments you plan to use and should document 
your ability to access clients, collect good quality data, and check on the quality of data. 

How mental health outcomes are conceptualized often depends on the perspective as shown on 
page 28, A Tripartite View of Mental Health Outcome. Different stakeholders (client/family, 
professional, and society) have different views of what they think is important to look at. 
One domain is psychopathology. From the perspective of the client and family, a reduction in 
problems and stresses are what they want to deal with. Professionals have categories for 
describing children, and they look for a decrease in symptoms, severity of diagnosis, and 
treatment needs. From the societal perspective, the salient outcomes relate to how restrictive 
the treatment is, how expensive the treatment is, where treatment is provided, length of stay, 
and how the child deals with the surrounding environment in the community. 

Another more positive domain is social competence. From the client's perspective, this 
involves an examination of self-esteem or how the child thinks about himself or herself, their 
sense of what resources are available. A professional would judge things like level of 
functioning and self-esteem, and on the societal side one could look at outcomes like school 
performance, absences, and teacher ratings. 

A third outcome domain of interest is satisfaction. From the client/family perspective, this 
would involve assessing their satisfaction with services. Professionally, this would involve 
looking at whether services meet professional standards or the quality of services. In terms 
of society, this involves looking at whether services meet regulations, licensing standards, and 
the like. 
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A TRIPARTITE VIEW OF MENTAL HEALTH OUTCOME: GOALS OF SERVICE 









i 


Domains of 
Functioning 




Stakeholders 




Client/Family 


Professional 


Society 


Psychopathology 


Decrease in 
referral problems , 
stress 


Decrease in 

symptoms , severity 
of dx, treatment 
needs 


Less expensive/ 
restrictive 
treatment (LOS 
down) , decreased 
# disruptive 
community 
incidents 


Social competence 


Feel better/more 
in control i more 
resources available 


Improved self* 
esteem , adaptive 
functioning 


Better school 

attendance, 

grades 


Satisfaction 


Satisfied with 
services 


Meets professional 
standards 


Treatment meets 
standards i 
regulations 
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One of the more recent techniques used for measuring clinical outcome is the structured 
clinical interview. A sample of four such interviews are shown on page 30. This is not 
meant to be an exhaustive list; there are other structured interviews available. These 
interviews are practically the only research-oriented way to obtain a DSM-III-R diagnosis that 
there is some degree of trust in, although you may not need to have DSM-III-R categorization 
in your project. The chart displays a number of items about each structured interview -- how 
long it takes to administer, the age range, type of interviewer required, whether it has a 
parent form, whether it has a training manual, how it deals with individual symptoms, summary 
scales, and diagnoses. The chart also includes information about reliability and validity and 
the availability of computer scoring. Included on the chart are the Diagnostic Interview for 
Children and Adolescents (DISC), the Child Assessment Schedule (CAS), the Schedule for 
Affective Disorders or Schizophrenia in school age children (K-SADS), and the Diagnostic 
Interview for Children and Adolescents (DICA). 

The data from structured interviews that people tend to use as outcome measures are the 
symptom severities that provide a sensitive way of looking at the children's perspective. This 
is the most time consuming aspect of data collection for children and parents, but they 
provide the most detailed, in-depth information. 

Another approach to outcome measurement, which is much less time consuming and easier to 
administer, is the use of a behavior rating scale. The chart on page 31 provides information 
about five behavior rating scales in terms of number of items, administration time, appropriate 
age range, availability of a training manual, who does the ratings, availability of norms, 
availability of scoring software, reliabilities, etc. The chart includes the Conners Parent 
Rating Scale (CPRS), the Conners Teacher Rating Scale (CTRS), the Child Behavior Checklist 
(CBCL), the Teacher Report Form (TRF), and the Behavior Problem Checklist (BPC). Thus, 
there are a variety of checklists available that are basically self-report. There may be a 
child, parent, and teacher form available which enables you to collect multiple indices. They 
are empirically-based instruments in that they are based upon lots of questions asked to 
children, parents, and others and research to discriminate between clinical and nonclinical 
populations. They do not provide DSM-III-R categories but rather provide other subscale 
information about the child. All these instruments are under constant refinement, so you 
should be cure to obtain the latest version. 

There ire fewer measurement instruments in the social competence area. The CBCL teacher 
form has a social skills subscale, and instruments dealing with level of functioning come under 
this category. Indicators of level of functioning typically are rating scales to be completed by 
the child's therapist. In addition, an interviewer who conducts a structured interview and 
collects a lot of data may complete a level of functioning scale after the interview. The 
Children's Global Assessment Scale provides current level of functioning information, and 
Vanderbilt University has adapted an adult functional assessment scale for use with children 
(Child and Adolescent Functional Assessment Scale - CAFAS). Another example of an 
instrument in this area is the Self-Perception Scale which examines the child's self-concept in 
areas of scholastic competence, social acceptance, physical appearance, behavioral conduct, and 
additional dimensions for adolescents such as romantic appeal and job competence. The child 
or adolescent completes this Instrument, and it may be used to show responsiveness to 
treatment since as they feel better about themselves they may change how they rate 
themselves. 

Family functioning is another relevant area, although measurement in this area has been very 
difficult to develop. One can see a family as an organization, a small one, but one that 
contains some of the same characteristics as larger organizations - complexity, tasl. 
orientation, and communication patterns. To assess families, it is possible to use observational 
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STRUCTURED INTERVIEWS FOR CHILDREN 





DISC 


CAS 


K-SADS 


DICA 


Number of Items 


2o4 


4 

to 


100 

A w w 


60 


Administration Time 


40-60 minutes 


45-90 minutes 


45-60 minutes 


40-60 minutes 


Age Range 
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6-16 years 


Interviewer 


Clinician or 
Layperson 


Clinician or 
Layperson 
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Clinician 


Clinician or 
Layperson 


Parent Form 


Yes 


Yes 


Yes 


Yes 


Training Manual 


Yes 


Yes 


No 


No 


Individual Symptoms 
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onset* curat ion r 
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Presence/absence- 
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11 content areas 
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DSM-III 


DSM-III*** 


DSM-III 
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high* 
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moderate** 


Discriminant Validity 
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high* 
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Concurrent Validity 
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Computer Scoring 


Yes 


Yes 


Yes 


Yes 


*High: .66< r .< 1.00 


**Moderate! .45 , 


< r > .65 ***CAS-R available for DSM-III-R 

***DICA available for DSM-III-R 



BEHAVIOR RATING SCALES 





CPRS 


CTRS 
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TRP 


BPC 


Number of Items 


93 


39 


138 




89 


Administration Time 


10-15 min. 


5-10 min. 


15-20 min. 


15-20 min. 


10-15 min. 


Age Range 


6-14 yrs. 
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5-13 yrs. 


Training Manual 
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Rater 


Parent (s)/ 
Caretaker (s) 


Teacher (s) 


Parent (s)/ 
Caretaker (s) 


Teacher (s) 


Parent ( s) , 
Teacher ( s \ , 
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with child 
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(4-16 yrs.) 
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Scoring Software 
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Yes 


No 


Treatment Sensitive 


Yes 


Yes 


Yes 


No 
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Interrater Reliability 
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(Teachers) 
.55 - .93 
(Parents) 
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ratings, structured interviews, observation of families accomplishing tasks, or more 
standardized questionnaires about family functioning. These latter self-report measures provide 
the perceptions of family members about what problems they have and how sensitive they are 
to these problems. Global family functioning can be looked at as a direct outcome of some 
projects, or i.i some cases a family measure may be used more as a mediating or moderating 
variable in the analysis. Thus, it may be important to collect information about the family 
even though they may not be a direct recipient of services. There are a number of potential 
instruments including the Family Environment Scale; the Family Assessment Device which deals 
with problem solving, communication, roles, and effective responsiveness; the Family Index of 
Regenerativity and Adaptation (FIRA) which looks at seven indices of family functioning such 
as social support, stress, resources, and level of distress; the Family Inventory of Life Events 
(FILE) which is a scale that deals with divorce, separation, stress, woik and family 
relationships, illnesses, etc.; and the Family Resource Scale which looks at resources that the 
family has available including fmancial support, social support, time, etc. 

Another area that may be important to look at is parental pathology because there is some 
indication that this mediates childhood illnesses and may explain why you have differential 
effects with different children. There are a number of clearly structured interviews that exist 
for use with adults, and there are scales like the Brief Symptom Inventory (BSI), the SCL-90- 
R, and other things on the market that can be used. 

There are no standardized measures to measure satisfaction; everyone invents their own which 
can be problematic. Typically, people rate services very high on satisfaction, particularly on 
global satisfaction. People tend to be so glad to get some sort of service that even poor 
services are something they like. There is an instrument called the Consumer Satisfaction 
Questionnaire, an 18-item scale that looks at satisfaction with services. Dr. Bickman stressed 
that in assessing satisfaction, this issue of specificity is critical. When you are very specific, 
the information you obtain is much more valuable than global measures of satisfaction because 
it pinpoints where the problems are. 

In addition, there is information to be collected about the background of the child. Who the 
child is living with, previous services received, and the like are important questions to ask. 
At Duke University, an instrument called the CASA is being developed to describe the services 
that a child has received. Because children typically move around so much, this information is 
often not obtainable from records. Further, there are behavioral indices that may be 
important to collect. School performance data (grades, attendance, disruptions at school), 
criminal justice data, and the like is important, although access might be difficult since some 
of these records are highly confidential. This is very powerful to policymakers. Because 
wkat does anyone know about a change in symptom level from 148 to 137 being statistically 
significant? We don't know what it really means. Or satisfaction changing from 4.7 to 5.2. 
The one thing that we do have in terms of being able to communicate with each other, are 
things like whether the number of days the child has been in school has increased, whether 
grades have improved, whether they are being arrested as frequently or hospitalized as 
frequently. A lot of these types of measures are very important and can be easy to collect in 
some circumstances." 
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INSTITUTIONAL REVIEW BOARDS 



Leu Bickman, PhJX, Director 
Center for Mental Health Policy 
Vanderbib University 

Almost every university has an institutional review board (IRB) that reviews proposed research. 
These boards vary individually with respect to requirements for different types of research. 
Research with children is much more sensitive than research with adults, and therefore, IRB 
requirements may be more stringent. One of the areas of concern is the issue of informed 
consent. Typically, you will be required to obtain permission from parents and in many cases 
from the child as well in order to involve them in your research. This usually requires the 
parent and adolescent to sign a consent form indicating that they understand the risks and 
benefits of the research and agree to participate. They also need to give informed consent to 
allow you to have access to their records, and informed consent also is required for any audio 
or video taping. 

IRB approval also is needed to collect data. If you will have access to records that contain 
personal identifiers, and there is information in the records that could harm individuals if it 
were disclosed, then a full review is needed. States may have IRBs as well as universities, 
and approval from both the state and the involved university may be necessary depending upon 
the circumstances of each proposal. Some IRBs require letters indicating cooperation from all 
institutions involved in the study before they will grant approval. These institutions may have 
their own IRBs or internal review boards, and so the research protocol may require approval 
by a group of people at each involved institution before the state or university will grant 
approval Thus, it is important to begin this review process early. The document "Protection 
of Human Subjects/ included as Appendix E, contains special regulations for research dealing 
with children which specify that the level of risk determines the type of informed consent and 
permission that are needed. 
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Research on Service Delivery and Systems of Cares 
Recommendations for the NIMH Child Mental Health Research Plan 
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Purpose and Scope 

The National Institute of Mental Health (NIMH) is currently in 
the process of preparing a children's mental health research 
plan. This plan will build on the recent report of the Institute 
of Medicine (IOM) of the National Academy of Sciences, and 
additional input provided to NIMH. 

The specific purpose of this paper is to aid in creating the NIMH 
research plan by providing recommendations on service delivery 
and systems of care research in children's mental health. The 
paper is an expansion of the IOM report in this area, and is a 
response to a request by the Director of NIMH for recommendations 
on service delivery and system of care resr-ich. 

The report is prepared by the Research and Training Center for 
Children's Mental Health at the Florida Mental Health Institute, 
University of South Florida. It is based on information received 
from a broad range of individuals in a short-time period. This 
paper is intended to identify critical issues and themes for 
research in this area, and discuss resources and funding 
mechanisms needed to support this research. It is neither 
intended to be a comprehensive review of existing knowledge in 
this area, nor a detailed long-term plan for future research. 

Further, this report is intended to specifically identify the 
opportunities and needs for research in service delivery and 
systems of care for children's mental health. The resources that 
this report identifies as being needed are intended to be new 
resources. They are above and beyond existing resources for this 
important area and also above and beyond the resources already 
being used or requested for such important areas as clinical 
research, the Child and Adolescent Service System Program 
(CASSP) , services research for adults, and general support of 
clinical and research training. 

Background. 

Service Development; At the outset of the 1980s, as documented 
in a report done for the Children's Defense Fund by Jane Knitzer, 
the status of services for children with emotional disorders and 
their families was grossly inadequate. In most communities the 
only service option available was infrequent, office-based 
outpatient treatment or hospitalization, with the consequence 
that there was an excessive reliance on hospitalization. Many 
states did not have a single full-time children's mental health 
staff person in their state office, only seven states had made 
significant progress towards developing a continuum of services, 
and collaboration between public agencies to meet the needs of 
children was the exception rather than the rule. 
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Since that time, assisted greatly by NIMH, and particularly 
NIMH's CASSP initiative, there has developed a tremendous 
increase in interest in the needs of children with emotional 
disorders and their families. States have strengthened their 
capacity to provide leadership in this area, new service models 
have been developed that provide intensive services in home and 
community-based settings, models of a system of care have 
emerged, the role of parents has been re-defined, and 
inter-agency collaboration has become more common. In response 
to the Mental Health Planning Act (P.L. 99-660), NIMH has taken a 
clear policy stance emphasizing the need for comprehensive 
community-based systems of care with a special focus on those 
children and adolescents with the most serious emotional 
disorders. In response to this, states have prepared plans that 
describe the specific system they intend to develop. 

Despite the fact that much progress has been made, as the results 
of a recent study of children's mental health by the U.S. Office 
of Technology Assessment indicates, "The majority of children 
with mental health problems fail to receive appropriate 
treatment. Many of the 6 to 8 million children in our nation who 
are in need of mental health interventions receive no care; other 
children, perhaps 50% of those in need of treatment, receive care 
that is inappropriate for their situation" (Saxe, 1987) . Part of 
the inadequacy of care clearly relates to lack of resources; 
another major factor, however, is the inadequate knowledge base 
both about the causes and treatment of emotional disorders, and 
the mechanisms for establishing and maintaining responsive and 
effective service delivery systems. 

At the same time that the public mental health sector has been 
emphasizing the need for community and family-based systems of 
care, the 1980s have been a time of rapid and enormous growth in 
free-standing psychiatric hospitals for children. For-profit 
hospital chains have expanded around the country, and the number 
of admissions to these hospitals for children and adolescents has 
increased dramatically despite the lack of evidence about their 
effectiveness (Burns & Friedman, 1989; OTA, 1986). 

Recent summaries of epidemiological research on point prevalence 
of emotional disorders have come up with estimates in the general 
range of 14% to 20% (Brandenburg, Friedman, & Silver, in press; 
Costello, in press) . As the IOM report indicates, "The magnitude 
of the problem of childhood mental disorders is only partially 
reflected in epidemiological data about prevalence and incidence 
of disorders. Substantial evidence suggests that many childhood 
disorders extend into adulthood either because of their chronic 
course or because of their adverse effects on the child's 
development (Kazdin, 1989). Moreover, enormous financial and 
social costs can be associated with mental disorders in children 
and adolescents." While making this point, the IOM found it 
difficult to quantify the actual financial and social costs at 
this time because the needed information is not available. 

Developments in Services Research; Given the human and financial 
costs of disorders in children, given the scope and magnitude of 



9 

ERIC 



51 



the problem, and given the increased interest in improving 
services, there is now and has been a tremendous need for 
research on service delivery and systems of care. This is 
particularly the case because of the special dependent status of 
children, the wide range of needs that they have and the complex 
web of statutes, policies and agencies that have been created to 
respond to these needs. Yet despite this need, until the last few 
years there has been almost a total absence of research on 
service delivery and systems of care. 

It should be noted in this regard that research on service 
delivery and systems of care in mental health is a new area, 
having a much briefer history than the longer established 
traditions of basic and clinical research. In fact, not only is 
it relatively new but it had its origins primarily in providing a 
"real world" test for new developments emanating from clinical 
research. However, there have been at least four significant 
developments at a service and policy level that have recently 
converged to create a strong need for more and better services 
research. 

First, the primary locus of treatment in the mental health field 
has moved from hospitals to community settings. This has created 
new challenges in understanding how to develop effective 
community systems. Second, public mental health systems have 
identified as their top priority the improvement of services to 
individuals with severe and persistent problems. Many of these 
individuals require services from a variety of different kinds of 
agencies over an extended period of time, thereby creating a need 
to take a multi-agency look at system development. Third, the 
role of states in planning and administering mental health 
services has grown with the implementation of the Alcohol, Drug 
Abuse, and Mental Health Block Grant program in 1980. This 
provides a more centralized responsibility for policy-making and 
planning. Fourth, there has been a rapid escalation of costs for 
health care in general, including mental health care. As a 
consequence, there has developed a need for research on such 
issues as approaches to cost containment, and the impact of 
different reimbursement strategies on service delivery. 

Among the impediments to research on service delivery and systems 
of care in children's mental health have been the lack of 
researchers trained in this area, and the lack of opportunities 
for those with interest and skill. Despite this, however, the 
second half of the 1980s have seen an increase in such research. 
Positive results from the NIMH Public — Academic Liaison (PAL) 
initiative, announced by NIMH in 1988, were seen in some states 
very rapidly in the form of increased collaborations between 
university-based researchers and policy-makers. In February, 
1988, the first national conference on services research was held 
on the theme of building an empirical base for service 
development. The third such conference will be held in February, 
1990, and the number and quality of research submissions has 
steadily gone up for these conferences. In keeping with the theme 
of the NIMH PAL initiative, these conferences have brought 
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together a combination of university-based researchers and public 
policy-makers. 

In December, 1988, NIMH issued a targeted research announcement 
calling for proposals dealing with mental health services for 
children. An indication of the interest in this type of research 
is that 29 proposals were submitted, with eight receiving 
funding. In identifying their priority concern, State Mental 
Health Representatives for Children and Youth included the need 
for better research and evaluation as a high priority, and 
clearly state policy-makers have demonstrated a receptiveness to 
collaborating with university-based researchers. Further, 
technologies for conducting service system research, while still 
in need of further development, have made significant progress, 
particularly in critical areas such as outcome measurement. 

All signs suggest that there is both a need and a unique 
opportunity for research on service delivery and systems of care. 
On the one hand, there is growing interest within the academic 
research community. On the other hand, policy-makers and 
advocates, working together to build the most effective systems 
that they can amidst rapid change, are eagerly seeking a strong 
research foundation for the system. 

Method 

Input for this report was provided in several ways. The major 
intent was to gather ideas from a broad range of individuals 
within a short-time period. This includes researchers involved 
in services research from a variety of academic disciplines, 
state-level policy makers in children's mental health, 
researchers within state departments of mental health, parents, 
advocates, representatives of national organizations involved in 
policy development, and administrators of local mental health 
programs . 

First, a meeting of 15 researchers, policy-makers, and advocates 
was held in Alexandria, Va. on October 3, 1989 to discuss major 
issues for which research is needed, and resources and funding 
mechanisms to support such research. Second, in order to expand 
the number and range of people consulted, a series of in-person 
and telephone interviews were conducted with approximately 25 
other individuals from around the country. Third, a draft paper 
was distributed to a group of 20 individuals for their comments 
and recommendations. Fourth, the draft paper was reviewed at 
the meeting of the Advisory Board for the Research and ""raining 
Center for Children's Mental Health in Tampa, October la and 16, 
1989. Altogether, as indicated in the list in the Appendix, over 
50 individuals participated in the process of developing this 
report in a short time period. 

Recommendations 

This section will first identify the substantive areas 
consistently identified as being in immediate need of research 
attention. These are not only important areas of need but also 
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represent areas in which the technology is sufficiently advanced 
and the manpower is available for a substantial contribution to 
be made. The section will then discuss funding mechanisms to 
support such research. This will lead to a discussion of 
research training needs in the field, and finally, to a specific 
recommendation of the financial resources needed to support the 
proposed activities. 

Focus of Research; Although many potential focuses for research 
were identified during this process, it is striking that there 
was significant consensus in the identification of priorities by 
participants in the process. Eight major themes, which are 
clearly overlapping, were identified and are presented here 
without any attempt to further prioritize them. 

1) Research on the Effectiveness of Strategies to Provide 
Alternatives to Residential Treatment through Comprehensive 
Community-Based Systems— In many states, currently two-thirds or 
more of the children's mental health budget is allocated to 
residential or hospital treatment. As the Invisible Children 
Project of the National Mental Health Association reports, about 
5,000 children are placed out of their own state each year in 
residential treatment facilities. Given the scarcity of dollars, 
the high cost of residential and hospital treatment, and the lack 
of research to document its effectiveness, many state 
policy-makers and advocates are seeking to develop systems of 
care that provide cost-effective alternatives to residential 
treatment. States and communities have developed a range of 
approaches to this problem, including modifying the array of 
services provided, the manner in which the service delivery 
system is organized, the reimbursement mechanisms, and 
decision-making processes about the need fo^ residential or 
hospital treatment. Given the policy emphasis placed within many 
states and within NIMH (e.g, P.L. 99-660) on the need for systems 
of care, there is a parallel need now for research that will use 
the overall community-based system as the unit of analysis rather 
than focusing only on specific service components or procedures. 
The Mental Health Service Program for Youth of the Robert Wood 
Johnson Foundation provides a natural and rich opportunity for 
research of this type on community-based systems. This is an 
area of research that is very complex and clearly calls for a 
strong public and academic collaboration. An important theme 
within this research, as well as the other research topics, 
should be the tangible financial impact and the overall social 
and emotional impact of the services on families as well as on 
children. 
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2) Research on the Effectiveness of Innovative Service 
Approaches — The 1980s have seen the expanded use of a range of 
innovative services for children with serious emotional disorders 
and their families. These include intensive home-based services, 
therapeutic foster care, respite care, family support, day 
treatment, mobile crisis services, case management, and 
individualized treatment using flexible funds. For some of these 
newer approaches there is an accumulating body of program 
evaluations that suggest their effectiveness. For the most part, 

54 



6 



however, the research base on the overall effectiveness of these 
approaches In relation to other services, and on the 
effectiveness of specific models is inadequate. In addition, 
this is an area that provides rich opportunities to integrate 
clinical research with services research by examining the 
effectiveness of particular approaches with youngsters whose 
problems are well-defined and to examine both quantitative 
outcomes and quality of service. As with the first area 
mentioned above, there is also clearly a need for the research on 
these services to have a strong longitudinal follow-up component. 

3) Research on the Effects of Different Reimbursement 
Mechanisms and Funding Strategies — Within recent years there has 
developed an increasing awareness of the impact of reimbursement 
mechanisms and funding strategies on service delivery. For 
example, in testimony before a congressional committee the senior 
author of the U.S. OTA report indicated that, "The reasons for 
the present inefficient and ineffective system are many, but one 
is increasingly central: Our methods for paying for mental 
health care. Rather than children's needs being paramount in 
deciding whether and whr.t type of treatment will be proffered, 
treatment decisions are increasingly drivr- by the health care 
reimbursement system" (Saxe, 1987) . There are a variety of 
issues that are of critical importance under this general 
heading. These include strategies for developing benefit 
packages both within the public sector (e.g., Medicaid and 
Champus) and the private sector (e.g., major private insurance 
carriers) that will provide incentives for use of home and 
community-based services, the advantages and disadvantages of 
managed care approaches, the actual direct and indirect costs of 
emotional disorders to families, public agencies, and third-party 
providers, the impact of different state funding strategies on 
service utilization at a local level, and strategies for 
providing quality care to children and families without any 
insurance coverage. Despite the importance of these issues, and 
other related financing topics, this has essentially been a 
dormant area of research in the children's mental health field. 

4) Research on the Effect' eness of Approaches to Involving 
and Supporting Families — During the 1980s there has been 
increased recognition of the great burden that is created for 
families of youngsters with emotional disorders. The sacrifices 
that families make are often overwhelming and these can quickly 
become public problems as well. For example, it is not unusual 
for families to become bankrupt because of the costs of care, 
many times parents must quit their jobs or reduce their work 
hours in order to fill the gaps in care between fragmented formal 
programs, and stresses related to caring for children with 
emotional disorders frequently contribute to marital problems 
(Friesen, 1989) . There has also developed a greater recognition 
of the potential benefits to families and children of providing 
services that are not only family-based but supportive of 
families. These developments have occurred, however, largely 
without the support of research to examine the relative efficacy 
of different family support approaches, the impact of different 
strategies for involving families in case planning and 
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decision-making as partners with professionals, barriers to care 
from a family perspective , and the effect of different 
organizational structures and practices on the responsiveness of 
service systems to the needs of families* Important issues such 
as the effect on families of parents having to relinquish 
custody of their children in order to secure special services are 
in great need of study. In addition, rather than the family 
perspective serving as an important beginning point for 
conceptualizing research, it has been virtually ignored. There 
remains now a major challenge to incorporate the family 
perspective both in establishing overall research agendas and in 
designing specific research projects. In addition, there remains 
also a need to assess the impact on family members of having a 
child with a serious emotional disorder, and the impact of 
various services and policies on family members. 

5) Research on the Effects of Multi-Agency Collaboration in 
Identification, Referral, and Treatment of Children with 
Emotional Disorders — There is a consistent body of research that 
suggests that children with emotional disorders tend to be 
under-identified as needing help by primary care physicians, 
educators, and child welfare staff, for example, while there has 
been some research on this topic, there is clearly a need for 
additional research to identify barriers to proper identification 
by non-mental health professionals, and to assess strategies for 
strengthening appropriate identification and referral, while the 
issue of identification is important, another critical issue for 
research is strategies for producing effective collaboration in 
actually serving youngsters who may already be receiving services 
from more than one agency, whose problems require services from 
multiple agencies, or who are dually diagnosed. Particularly 
amongst children with the most serious emotional problems, there 
is almost inevitably a need for services from agencies other than 
mental health, be it special education, child welfare, juvenile 
justice, substance abuse, or primary health care. The 
inadequacies of any single system add to the need for services 
from the other systems. In recognition of this, across the 
country there have developed a number of strategies for jointly 
funding services, for combined case planning on individual 
youngsters, for clarifying roles and responsibilities of 
different agencies, and for promoting closer collaboration. Yet 
despite the importance of this area, and the enormous range of 
activities that have been initiated to improve collaboration, 
there is little systematic research either on strategies for 
achieving such collaborations, or their actual effects on 
children and families. 

6) Research on Services to Minority Children and their 
Families — A major focus of the NIMH CASSP initiative is to work 
towards the development of a system of care in which services are 
geared to the varied needs of the culturally diverse population 
in the United States. As our population becomes increasingly 
more diverse, the need to develop and evaluate different 
strategies for increasing the flexibility and cultural 
appropriateness of services for minority children and their 
families grows as well. Other research needs include studying 
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the differential treatment within the mental health system of 
children and families from different ethnic and cultural 
backgrounds, assessing the effectiveness of policy and training 
strategies to produce a system that is more responsive to 
minority children and their families, and examining the different 
manifestations of pathology in different cultures. It is 
recognized that within NIMH there already exists a focus on 
research on minority issues in mental health; at this point there 
is a great need for increased emphasis, particularly as pertains 
to the organization of service delivery systems. 

7) Research on the Organization of Services— The 
organization of a service system to provide services to children 
with emotional disorders and their families is highly complex. 
There are many variations in such issues as the nature of 
responsibilities of different state agencies, the relationship 
between the state and local levels, the relationship between the 
judicial and executive branches of government, and the existence 
of particular formal or informal structures to promote 
inter-agency collaboration. Even within different school systems 
in individual states there is considerable variability, despite a 
common federal policy structure on education of children with 
handicaps. A recent trend within states is to explore the 
advisability of re-structuring their service system to create a 
consolidated children's agency that provides mental health 
services as well as other services. Yet at this time there is 
very little research to guide policy-makers as they address this 
and other very important but complicated organizational issues 
involving different levels of government and different systems of 
service. 

8) Research on Assessment, Case Planning, and Clinical 
Decision-Making — The manner in which information is gathered 
about children by service providers, treatment plans are 
developed, and decisions are ultimately made is very important 
and also very under-studied. The Invisible Children Project of 
the National Mental Health Association found that most states had 
no formal, standard practice that they followed in deciding 
whether youngsters should be placed in residential treatment. 
Amongst those states that do have such a procedure, the 
procedures are considerably varied. Further, although there has 
been considerable progress made in clinical assessment under 
controlled conditions and in innovative projects, there is almost 
no information about the state of assessment practices around the 
country, or the impact of different practices. Further, creative 
case planning has been identified as a key requirement for 
effective service provision but there is very little known about 
case planning as it is currently practiced, or as it should be 
practiced. This is an area of research in which the 
opportunities and the need are both clearly there. 

Methodological Research— In addition to these eight 
substantive areas, an important need for more methodological 
research was identified with two particular focuses. The first 
focus was the need to develop improved methodologies for 
describing and measuring community-based systems of care. The 
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absence of such methodologies impedes progress in research in 
which the basic unit of analysis is the system of care rather 
than smaller components of it. Given that the policy focus both 
at a federal and state level is on the development of 
community-based systems of care, it is imperative that more 
methodological research be done in this area. Such methodologies 
would need to be able to describe the formal structure of systems 
(e.g., the existing agencies, funding mechanisms, decision-making 
processes, inter-agency agreements, statutes), and the actual 
workings of the systems (e.g., the actual identification and 
referral practices, the nature of the interactions between 
agencies, the development of service plans, and the current 
patterns of care seeking) . 

The second focus was the need to refine methodologies for 
assessing cost-effectiveness and cost-benefit of services. 
Although progress has been made in this area, the complexity of 
determining precisely which costs should be included, how those 
costs should be calculated, how indirect costs should be 
assessed, and how different outcomes should be compared to costs 
still presents a major problem. While there is a clear consensus 
on the importance of research that looks at cost-effectiveness 
and cost-benefit of services, there is also agreement on the need 
for more methodological research to advance the field. 

Since research on service delivery and systems of care is 
ultimately intended to result in an improved system of services, 
there is a special need to emphasize the importance of 
supplementing traditional approaches to dissemination (e.g., 
publications in refereed journals, presentations at professional 
conferences) with non-traditional methods. Policy-makers are a 
major consumer of this type of research, and while they appear 
eager to get the new information the traditional methods of 
dissemination are generally ineffective in reaching them. It 
therefore becomes important that as part of an increased emphasis 
on service delivery and systems of care research there be a 
parallel emphasis on disseminating information more directly to 
planners and policy-makers. This can be done through such 
mechanisms as brief, non-technical publications distributed 
directly to the target audience, and through presentations at 
meetings traditionally attended by planners and policy-makers. 

Overall, as the last two methodological issues illustrate, it is 
important to emphasize that in research on service delivery and 
systems of care the precise specification of variables and 
measurement procedures are intrinsically extremely complex. 
Also, when the unit of study is some aspect of the service 
system, the opportunity for the researcher to control all of the 
critical variables is not as great as in basic or clinical 
research. In addition, from a developmental standpoint this field 
of research specialization is still in its early stages of 
growth. While these conditions create special challenges for 
researchers in this fie. " they are not insurmountable and there 
is no question of the n A for more research of the type 
described here if a strong empirical foundation for service 
development is to be established. Both in terms of its 
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understanding of the substantive issues and its ability to deal 
with the methodological challenges, the team of academic 
researchers working in combination with policy-makers, 
agency-based researchers, parents, and advocates appears ready to 
develop research proposals that will significantly advance the 
field. 

Funding Mechanisms; There are a number of different funding 
mechanisms for support of system of care and service delivery 
research, most of which already exist, which are needed. These 
include the following: 

1) Individual Project Grants— This is the primary funding 
mechanism for NIMH research. This is an important and 
appropriate mechanism which needs to be expanded in order to more 
adequately meet the need in the field and capitalize on the 
present opportunities. In addition to the expansion of this 
research support mechanism, it is important to insure that there 
be an ongoing peer review committee composed primarily of experts 
in services research in the children's field. Given the 
complexity of this field and the differences between it and the 
adult field, a fair, knowledgeable and consistent review process 
requires a standing multi-disciplinary committee specifically for 
children. Such a committee was constituted on a special basis 
for the recent special announcement issued by NIMH (December, 
1988) for services research in child mental health, and a similar 
committee should now be established on an ongoing basis. 

2) Center Grants — The mechanism of funding research centers 
is particularly important in this field given the need for teams 
of researchers from both traditional mental health disciplines 
and other disciplines (e.g, economics, sociology, education). 
NIMH currently funds five centers for research on the 
organization and financing of services for the seriously mentally 
ill. While the center model as utilized by NIMH is a valuable 
one, and several of these already- funded centers are initiating 
projects for children, children's research represents a small 
part of their agenda. There is a new for capitalizing on this 
important mechanism by funding research centers specifically on 
service delivery and system of care research in children's mental 
health while at the same time creating new opportunities for 
existing centers to focus on children. 

3) Service Demonstrations— This is a mechanism that is not 
currently available through NIMH support in the children's mental 
health field although a recent grant announcement was issued for 
such projects for adults through the NIMH Community Support 
Program. This type of mechanism provides funding for service 
demonstration projects that include a strong research and 
evaluation component. Such a mechanism is particularly ideal for 
research on the efficacy of innovative service approaches, such 
as some of those highlighted by the NIMH CASSP initiative 
(Goldman, 1988; Stroul, 1988 & 1989; & Stroul & Friedman, 1986), 
for research on alternative approaches for supporting families, 
and for research on approaches to serving minority children and 
their families. The support of such projects is a logical 
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extension of the NIMH CASSP effort. It is important to 
emphasize, however, that it should be developed not as a 
replacement for existing CASSP activities but as an important 
research — demonstration extension of CASSP with additional funds. 

4) Small Grants— This grant program has recently been 
modified to allow for funding of grants for up to two years at a 
maximum of $50,000 per year. A special advantage of these grants 
is that the time for their review is only about four months. The 
relatively rapid turnaround time for these grants, and the new 
potential both in terms of duration and amount of funding makes 
this a valuable funding mechanism. It can be particularly 
helpful in the early stages of relationship-building between 
university-based researchers and public agencies by providing 
funding for small-scale projects that can later be expanded 
through other research support mechanisms. Also, there are 
growing opportunities in the field to advance knowledge by 
capitalizing on naturally-occurring experiments as important 
changes are made within service systems. This grant mechanism 
provides a means for public agencies and universities to 
collaborate in a relatively short-time period on meaningful 
studies to take advantage of the opportunities presented by hese 
changes. In order for this to be an effective research support 
mechanism, however, it is essential that there be not only a 
quick turnaround time but a review by a committee that is expert 
in services research. 

5) Inter-agency Research Support — NIMH is the federal 
agency with the strongest commitment to children's mental health 
and the greatest talent and technical resources to address this 
area. However, many other federal agencies share a strong 
interest in increasing the knowledge base for serving children 
with emotional disorders and their families. This includes 
groups within other ADAMHA institutes, the National Institute of 
Health, the Administration for Children, Youth, and Families, the 
Bureau of Maternal and Child Health, the Department of Education, 
and the Department of Justice. The field would greatly benefit 
from NIMH assuming a strong leadership role in developing a 
multi-agency services research agenda at the federal level, and 
developing mechanisms for multi-agency review and support of 
proposals. An important precedent for this is the joint funding 
of two research and training centers in children's mental health 
by NIMH and the National Institute on Disability and 
Rehabilitation Research of the Department of Education. 

In addition to these mechanisms, other important means of 
supporting research include research training grants and career 
development awards. Research training grants are discussed in 
the next section. There are two primary types of research career 
development awards with the general goal of enabling researchers 
to engage in research on a full-time basis over a period of time. 
One broad category is the individual career development awards 
that serve an important purpose in supporting researchers who 

have completed their training and are now seeking an opportunity 
to devote their full-energies to research over an extended period 
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of tine. The second type of award is the First Independent 
Research Support and Transition (FIRST) award which is more 
specifically designed to provided an initial period of support 
for a newly independent researcher. These awards may provide 
support of up to $350,000 for five years. 

Research Training! While it is clear that the field is ready for 
significantly expanded research in service delivery and systems 
of care for children, it is equally apparent that the future 
growth and development of this field requires a strong commitment 
to research training across a wide range of disciplines, and 
support of beginning researchers. This support of research 
training is required both at the pre-doctoral and 
post-doctoral level. This report is in full support of the 
emphasis with the IOM report on the importance of this area but 
goes beyond the report to specifically address research training 
for service delivery and system of care research. 

The specific mechanisms needed to support enhanced research 
training are varied. They include training grants to 
institutions for pre-doctoral and post-doctoral training, and the 
support of multi-disciplinary services research training centers. 
They also include and should emphasize training opportunities for 
minorities. 

It is essential that these awards for research training have a 
broader focus than the traditional mental health disciplines. 
They may conceivably include such disciplines as anthropology, 
sociology, economics, education, and health policy and 
management, for example. At a post-doctoral level, it is 
particularly recommended that research training centers be 
supported with a multi-disciplinary focus. Such 
multi-disciplinary services research training centers need not 
have an exclusive focus on children; it is essential, however, 
that they be required to have a significant focus on children. 

Funding Amounts: The previous sections of this paper have 
discussed the important themes within service delivery and system 
of care research for children, research support mechanisms, and 
research training. This section provides specific 
recommendations for funding amounts. These are offered for the 
different research and research training support mechanisms. 

Research Support — In response to its recently issued 
announcement for services research, NIMH received more proposals 
that were approved than it was able to fund. It actually 
provided approximately $2 million in funding. This 
announcement, along with other developments in the field such as 
the NIMH PAL initiative and the Robert Wood Johnson Foundation 
Mental Health Services for Youth grant program, continue to 
attract growing interest not only from university-based 
researchers but from teams of academicians and staff from public 
agencies. Further, activities such as the Robert Wood Johnson 
grant program provide excellent opportunities for research that 
extend beyond the evaluation that will be directly funded by the 
foundation. Given these new opportunities in the field, and the 
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response to the last special NIMH announcement, it is recommended 
that in Year One Research Support for Individual Project grants 
be $5 million. This should be expanded by $3 million per year to 
a total of $17 million by the end of the five year plan. 

In addition, it is recommended that NIMH support four research 
centers specifically for children during year one. Given the 
cost of conducting research on complex service delivery systems, 
it is recommended that they be funded during the first year at $1 
million each for a total of $4 million. In addition, to 
encourage that existing centers increase their focus on children, 
it is recommended that there be an additional $1 million to 
supplement funding of existing NIMH- supported research centers 
for the organizing and financing of services. This raises the 
total cost for year one for centers to $5 million. There should 
be an additional two centers added during year two and year three 
each for a total of eight centers plus supplemental funds to 
existing centers at a cost of $9 million. Clearly the Center 
mechanism is extremely important for supporting the development 
strong multi-disciplinary research teams with an ongoing program 
of research. The response to the previous NIMH center 
announcements have indicated that such announcements have great 
potential for facilitating the assembling of such teams, but 
primarily around the needs of adults to this point. 

Next is recommended that NIMH support ten service 
demonstration projects at an average cost of $400, 000 during the 
first year for a total cost of $4 million. This should be 
increased by five new projects per year for a total of 30 
projects and $12 million after five years. The response of the 
children's field indicates a readiness and eagerness to undertake 
service demonstration efforts, and the response to the service 
demonstration announcement for CSP shows a willingness and 
ability of the field to develop demonstration efforts with strong 
research components. There is an enormous need for such 
projects, particularly given the changing conf igaration of 
services and needs. For such projects to both have a strong 
research component and mount a meaningful service demonstration, 
there is a need for an average of $400,000 per project. It is 
important to emphasize again that this request is for new money 
for service demonstration projects, While these projects 
represent a logical extension of NIMH-aupporred CASSF activities, 
they should extend and not replace existing CASSP activities. 

In the small grant area, it is recommended that there be $1 
million in research support during year one. Thim win provide 
for 20 grants at $50,000 per grant. The small grant program 
provides an excellent opportunity to build strong 
academic — public sector linkages, ami to strengthen the existing 
knowledge by taking advantage of naturally occurring system 
changes. This mechanism will not only be well-utilized but will 
further the achievement of the objectives of the PAL initiative. 
These grants should be increased by 10 per year for a total of 50 
grants for $2.5 million after five years, 
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The next research support mechanism is inter-agency grants. The 
development of effective inter-agency support mechanisms is more 
time-consuming than the expansion of the existing mechanisms. It 
is recommended that during the first year NIMH utilize its 
leadership to develop an inter-agency research agenda along with 
appropriate funding mechanisms. No funds for actual research 
support are requested. However, it is recommended that during 
year two there be $3 million in research support for ten projects 
at an average of $300,000 per project, and that this be expanded 
by ten projects each year for a total of $12 million by the end 
of year five. 

It is recommended that 10 individual Career Development Awards in 
services research be awarded during year one at $100,000 each for 
a total of $1 million. This will provide important and 
much-needed support for researchers who have completed their 
training and are now seeking an opportunity to devote their 
full-time attention to their research career. Having invested in 
the training of these individuals, it is important now to provide 
mechanisms to support them in critical stages of their career. 
This should be increased by 10 awards per year until there are 50 
such awards by the end of the fifth year for a total amount of $5 
million. 

The final support category for research is FIRST awards (First 
Independent Research Support and Transition awards) . It is also 
recommended that $1 million in support be provided during the 
first year with this increasing to $3 million by year five. The 
average cost per grant in this category would also be $100,000. 

It should be noted that the recommendations here focus 
exclusively on external support of research. Clearly for the 
necessary expansion to take place there is a need for internal 
expansion within NIMH as well. In particular, there is a need 
for NIMH to add full-time staff with expertise in service 
delivery and systems of care research in children's mental health 
to provide technical assistance to potential grantees and to 
organize and administer the research programs. 

In summary, the following recommendations for research support 
are made: 



Funding Mechanism Year Qne Year FiVS 

Individual Project Awards $ 5.0 $17.0 
Center Awards $ 5.0 $ 9.0 
Service Demonstration Awards $ 4.0 $12.0 
Small Grant Awards $ 1.0 $ 2.5 
Inter-agency Awards 0 $12.0 
Individual Career Development $ 1.0 $ 5.0 

FIRST Awards S-JLJ! SJlxfl 

Total $ 17.0 $60.5 



Research Training— For research training the only general 
category to be included here is support of Institutional Training 
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Grants for both pre-doctoral and post-doctoral research training. 
This includes support of multi-disciplinary training centers for 
services research and support of research training for 
minorities. It is recommended that there be awarded 15 grants 
during year one specifically for services research at the average 
amount of $500,000 for a total of $7.5 million. This 
recommendation is based on the virtual complete absence of 
support at present for research training in this area. Given the 
importance of developing a strong corps of well-qualified 
scientists to do this important but complex research, it is 
further recommended that this be increased by six awards per year 
for a total of 41 grants and $19.5 million in year five. 

Summary: Overall, the recommendations call for support during 
the first year of $24.5 million of which $17 million is for 
research support and $7.5 is for support of research training. 
By the end of year five, these recommendations call for $60.5 
million in research support and $19.5 million in research 
training support for a total of $79.5. 

These figures, while reflecting a large increase over present 
spending, are modest in relation to the direct and indirect cost 
to individuals, families, and society of emotional disorders in 
children and adolescents. There are m«ny individual states, for 
example, that spend more public money per year just for 
psychiatric hospitalization for children from both state and 
federal sources than the total requested allocation for research 
and research training support. Further, these figures are even 
more modest considering the complexity of the service delivery 
mechanisms in the children's mental health field with the 
multiplicity of agencies and services involved. Clearly in 
virtually any other area of health care the expenditures for 
research on the service delivery mechanism would dwarf the 
comparable expenditure in children's mental health. Particularly 
as progress is made in the basic and clinical research areas in 
mental health, it will be tragic if the knowledge base in the 
service delivery and system of care area has not kept pace so 
that the field is ready to fully capitalize on th* new knowledge 
on behalf of children and their families. 

This is an unusual time in that recent years have seen much 
productive activity in the children's service delivery sector, 
state policy-makers are seeking empirical findings to assist them 
in their difficult jobs, parents and professionals are beginning 
to form powerful and effective partnerships, the technology of 
services research is making significant progress, and 
academicians and public administrators are sitting together to 
address problems of mutual concern. The proposed program of 
support for research and research training will be an important 
step towards responding to the enormous need, and capitalizing on 
the unusual opportunity that is present. 
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SIX IMPORTANT QUESTIONS 

Questions About the DEMONSTRATION 
1 • What do I want to change? 

2. How can I measure these changes? 

3. How can I intervene to produce changes? 

Questions About the RESEARCH DESIGN 

4. How can I detect any changes which do occur? 

5. How can I be certain that my intervention caused these 
changes? 

6. How can I ensure that others can learn from my success? 




QUESTION 01 - What do I want to change? 



Level of focus: 

* State- level system changes 

* Local- level system changes 

* Service delivery changes 

* Client outcome changes 



"Unit (s) of analysis": 

* Systems 

* Families 

* Children 

Spefrlic problem(s) to 'be addressed 



Magnitude of the problem(s) 
Trends 



QUESTION #2 - How can I measure these changes? 

Validity {measures what it is intended to measure): 

* Consistent with common usage 

* Consistent with alternative measures 

* Internally consistent 

* Predicts future behavior 



Reliability (produces same results repeatedly): 

* Composite measures better than separate measures 
t% * Multiple measures better than single measures 

* Uniform data collection techniques 



Sensitivity to change: 

* Range of possible responses 

* Continuous measures better than categorical ones 

* Guard against "ceiling" and "floor" effects 
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QUESTION §3 - How can I intervene to produce changes? 



Concept : 



* Theoretical base 

* Clear "program theory" of key elements (no 
"confounding" ) 

* Powerful intervention 



Implementation : 

* Feasible (experience, resources) 

* Consistent 

* Abrupt 

* Monitored 

* Documented 



Little "diffusion of treatment": 

* Resentful demoralization 

* Compensatory rivalry 

* Attempts to equalize treatments 



QUESTION H - How can I detect any changes which do occur? 



"Statistical power" depends o. * 

* Sample size 

* Effect size 

* Tests conducted 

* Alpha level 



Sample size: 

* Most common change 

* Can be hard to increase 

* Technical issue (see book review) 



Effect size: 

* Actual effect 

* Integrity of treatment (strong, consistently 
delivered, not diffused) 

* Data collection (consistent, sensitive, at maximum 
impact ) 
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Tests conducted: 



* Parametric better than non-parametric 

* Blocking, covariates, variables in regression 
equations (stratified sampling) 

* Repeated measures 



Alpha level: 



* "Legendary p<.05 " 

* Might be relaxed somewhat 



QUESTION |S - How can I be certain that my intervention caused 
these changes? 



Pre-post a common design: 

THEORETICAL RESULTS: 

PRETE8T-P0STTEST RESEARCH DESIGN 



IMMOVtD HHAAOft 




TIME 



But other possible explanations for this change ("plausible 
rival hypotheses"): 

* History (i.e., other events) 

* Maturation (i.e., growth) 

* Testing (i.e., familiarity) 

* Instrumentation (i.e., change in measuring 

instrument) 

* "Regression to the mean" (i.e., extreme scores 

tend to revert to average levels) 

* Selection (i.e., different populations in 

different groups) 

* "Mortality" (i.e., differential dropout rates) 
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Designs to reduce these "threats to validity": 

* Pre-experiroental (bad): 

* Performance monitoring 

* Pretest-posttest (one group) 

* Case studies 

* Cross-sectional surveys 

* Qua si -experimental (better): 

* Non-equivalent control groups (possibly with 
"matching" ) 

* Longitudinal time series (30+ data points) 

* Randomized experiments (best): 

* Pretest-posttest (two or more groups) 

* Post-test only (two or more group 




QUESTION #6 - How can I ensure that others can leara from mv 
success? 



Can the results be generalized: 

* To target population 

* Across other populations of interest 



Sampling plan: 

* Multiple populations 

* Stratified 

* Random sampling 
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Book Reviews 



4. sociO'technohgical dimensions -my a in which 
work should be organized 

5. theories of knowledge (epistomologies) 

The foundations for Kogan's model are selected 
from the theory of knowledge, social psychology and 
social policy and illustrate how various assumptions 
about educational issues can promote different forms 
of poetical control, education accountability being a 
wnique and significant case. Professor Kogan has pro- 
vided us with a comprehensive examination of the 
issues at hand and does a credible job in broadening the 
perspective of individual* examining the accountability 
issue. 

Although the context is almost exclusively British, 



the model has universal relevance for addressing the 
issue of education accountability and provides the 
reader with a comprehensive portrait of the interactions 
of the various elements of the model. Often educators 
and non-educators tend to focus on one particular 
aspect of accountability and fail to properly integrate 
the dimensions of value, affectiveness, sociotechnoiogy 
and epistomology. 

Professor Kogan's work is well written, succinct, and 
stimulating. The book would be valuable reading for 
grauuate students in education and practicing school 
administrators, who must face this issue almost daily. 
The slight inconvenience of adapting the concepts pre- 
sented to a local framework should not deter scholars 
from enjoying this analytic overview. 



ISow Many Subjects? Statistical Power Analysis in Research, by Helena Chmura Kraemcr and Sue Thiemann, 
Newbury Park, CA: Sage, 1987, 120 pages. 



Reviewer: Charles S. Rcichardt 

If you want to learn how to do a statistical power 
analysis with a minimum of effort, this is a book you 
should read. 

As commonly used, the label "statistical power anal- 
ysis" encompasses two separate, though related, tasks. 
One task is to determine the level of power that is 
obtained when a statistical significance test is used with 
a given sample size. The other task it to determine the 
sample size needed to obtain a given level of power. As 
the title of the book suggests, the focus of the volume 
h on the second task; determining the necessary sam- 
ple size, tn designing a study, this task is usually the 
more important of the two. Nonetheless, the text allows 
you to perform the other task as well. 

To perform either task, you need to do a few cal- 
culations and look up the results in a table. The neces- 
sary calculations are explained with the use of clearly 
labelled formulas and concise examples. Different sta- 
tistical significance tests require different formulas (and 
a couple in volve calculating an exponential or arcsint). 
But across the different statistical tests, the formulas 
are presented using a parallel structure. And most 
importantly, the formulas are easy to find and deci- 
pher, both in the text and in a summary table. As a 
result, after you have read the text once, months later 
you wilt be able to recover the appropriate formulas 
and figure out how to use them quickly and easily. 

After you have performed the relevant calculations, 
you then look up an answer in a power table. This step 
is also quick and easy because there is only one table to 
use and it is laid out in a simple fashion. You use this 
one table to determine the level of power that is 
obtained for a given sample size as well as to determine 
the sample size needed to obtain a given level of power. 
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You also use this one table for all the different statis- 
tical tests. Its all very simple. 

Of course, statistical power analysis is not completely 
mechanical. A thoughtful answer requires thoughtful 
input. For example, to determine the sample size to use 
in a study, you must decide, among other things, what 
level of power is adequate and how small a difference 
between the null and alternative hypotheses you are 
interested in detecting. But once you make all of these 
substantive decisions, determining the necessary sample 
size is purely computational. The beauty of the text by 
Kraemer and Thiemann is that the computations are 
made so simple conceptually that you can focus your 
thinking on the important substantive decisions. 

Most of the commonly used statistical tests are 
covered in the text. Specifically, the text shows how to 
perform power analyses for tests of means, differences 
between means, proportions, differences between 
proportions, correlations, bivariate regression slopes, 
and variance ratios. For most of these tests, the results 
of the power calculations are approximations, but there 
seems to be little need for more exact answers. Often 
the formulas that are provided allow for useful com- 
plexities such as unequal sample sizes, with little loss in 
conceptual simplicity. The text also presents procedures 
for a»essing power in less commonly used tests such as 
tests of Spearman's rank correlation and Kendall's tau. 
These tests axe not covered in the classic volume by 
Cohen (1977), or anywhere else that I know of. 

Other statistical significance tests, however, are not 
considered in the volume. For example, you will have 
to look elsewhere (e.g., Cohen, 1977) to learn how to 
do power analyses for tests of coefficients in multiple 
regression and for tests of diff erences between correla- 

?0 



Book Reviews 



tions. Kraemer and Thiemann do give intuitively appeal- 
ing procedures for determining power in a balanced 
analysis of variance and in two-way contingency tables. 
But the proposed procedures are a bit piecemeal and 
researchers may wish to supplement them with proce- 
dures described in other sources, such as Cohen (1977). 

In addition to explaining how to perform power cal- 
culations, the text also presents a number of principles 
for making research cost-effective. For example, the 
text demonstrates how power is lost by using statistical 
tests that dichotomize variables measured on a contin- 
uous scale. The text also demonstrates the loss in po*ei 
that results from using unequal sample sizes in tests* of 
mean differences, as well as the increase in power that 
results from increasing the variability of the indepen- 
dent variables in a regression analysis. 

Unfortunately, a couple of the principles that are 
presented in the text are incorrect. The text incorrectly 
specifies the condition; under which it would be advan- 
tageous to include a pretest measure in a between-group 
comparison. This is because Jhe text considers the value 
of adding a pretest only for an analysis of variance on 
pretest-posttest differences rather than for an analysis 
of covariance which would be more powerful under 
most conditions. Indeed, perhaps the most glaring 
omission in the text's suggestions for cost-effective 
design is the failure to emphasize the increase in power 
that can be obtained by Adding a covariate. The text 
also badly misinforms researchers about the value of 
blocking in analysis of variance. Nonetheless, the vir- 
tues of the text outweigh the errors. 

Another notable feature of the volume is that it con- 



denses its wealth of information into a remarkably 
small packaje. The text of the volume along with 
accompanying tables takes up only 81 pages. In addi- 
tion, the pages are small and the type is large. You can 
read cover to cover in two to three hours. To learn how 
to do a power analysis for a specific lest (e.g., a test of 
* difference between proportions), you would have to 
read only two chapters which would take only about 
half an hour; a small investment to pay for biing able 
to do a power analysis like a pre, 

All in all this is a wonderful little book. It makes 
power analyses quick and easy to perform which means 
that youll be more inclined to perform them. Though 
I think very highly of the volume, I must add one reser- 
vation About its use. 

If you are going iv> do statistical significance tests 
you ought to perfurm power analyses. But should you 
use statistical significance test* in the first place? Often, 
I believe the answer is no. This is because confidence 
intervals often should be used in place of statistical sig- 
nificance tests. Part of the reason is that statistical sig- 
nificance tests investigate only the direction or existence 
of a parameter while a confidence interval investigates 
the size of a parameter, and knowing size usually is 
preferable (Reiehardt & Gollcb, 1987). 

By explaining simply and clearly how to do power 
analyses, this book can greatly improve your use of sta- 
tistical significance tests. For this reason. ! hope you 
buy the book and use it. Although this volume can 
greatly improve your use of statistical significance tests, 
your research often could be improved more by using 
confidence intervals instead. 
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VERMONT SYSTEM FOR TRACKING CLIENT PROGRESS' 



INTRODUCTION and PURPOSE 

The purpose of this project is to develop a system for tracking client progress through 
the Vermont System of Care for Children and Adolescents. The information that is collected 
will be used to improve the provision of services within this System. The clients will be children 
and adolescents with severely malac usted behavior who are at risk of being placed in a more 
restrictive setting. 

One way to improve a system of care is to obtain reliable data that relates treatment 
procedures to treatment outcomes. At the present time, however, there are few service delivery 
systems that can empirically document behavioral characteristics of the different children that are 
being served, how the children are being served, and the level of behavior adjustment during and 
after services. When decisions are made to send a child to an out-of-state residential treatment 
program, or to wrap individualized services around a child in his or her community it is vers 1 
unlikely that the decision is based on data that show that the outcome for that treatment is 
better than that for any other treatment. 

There are several ways to defend making treatment decisions in the absence of 
empirically-based outcome data. It can be argued that 1) different treatments don't make a 
difference, 2) different treatments do make a difference but we know the difference without 
outcome data or 3) outcome data may improve our knowledge of the differences but it would 
not be cost-effective to obtain it. While there is little agreement with the first argument there is 
considerable support for the other two. With respect to the second, most case-workers/case- 
managers believe they know whether or not a particular treatment program is effective, at least 
with respect to the alternatives that are available. They make decisions that are "in the best 
interest of the child." Nevertheless, it is not uncommon for case-workers/case-managers to 
disagree amongst themselves. "I use program A because I've had pretty good experiences with 
it. M "She tends to use program B more than I do because she's had better luck with it." The 
opinions tend to be shaped by experiences with specific children. While personal experience is, 
and will always be an important part of any treatment decision, the question is whether 
treatment decisions can be improved with more comprehensive and more objective outcome 
data. Would it help to know how similar children responded to a range of different treatment 
programs over a two or three year period? In a few locations in Vermont, intensive, 
individualized wraparound services are being administered as an alternative to out-of-community 
residential treatment. Would it help to compare children who receive these very different 
treatments, both in terms of the nature and severity of maladjusted behavior before treatment as 
well as with adjustment outcomes after treatment? 



'This client tracking system is based on the work of John VanDenBerg (Alaska Division 
of Mental Health and Developmental Disabilities), David Born (University of Kansas), and v >hn 
Burchard and Mark Schaefer (University of Vermont). It is a collaborative effort between the 
Vermont Departments of Mental Health and Social and Rehabilitation Services, the Vermont 
Division of Special Education, the Department of Psychology at the University of Vermont, the 
Alaska Departments of Health and Social Services and Education, and the University of Alaska. 
Much of this work has been supported by the National Institute of Mental Health (Child and 
Adolescent Services System Program). The purpose of the copyright is to promote the 
coordination of any research involving the use of the data collection instruments used in this 
tracking system. 
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The Vermont System for Tracking Client Progress (VSTCP) is based on the belief that 
more comprehensive and more objective client outcome data can result in significant 
improvements in the existing System of Care. Also with respect to the third concern, it is 
believed that meaningful and useable outcome data can be obtained without creating an 
excessive burden in terms of costs and labor. As will be seen below, the most critical and labor- 
intensive component of VSTCP is obtaining adjustment data on a daily basis. To achieve that 
objective, it is necessary to identify the most important adjustment indicators and define them in 
such a way that they can be recorded reliably by a primary caretaker in one to three minutes 
each day. On the basis of extensive preliminary tests in both Vermont and Alaska, it appears 
that such an objective can be achieved. 

METHODS 

Note: The term "residential setting" will be applied to all living situations, whether the 
child is at home or in some other setting. The term "substitute-care" indicates that the 
child is living in a setting where the primary caregivers are individuals other than the 
child's natural parents or relatives. 

Subjects 

Children and adolescents with severe behavior maladjustment, ranging in age from 3 to 
22 years, will be placed on the Vermont System for Tracking Client Progress. It is anticipated 
that the project will add 50 clients to the tracking system for each of three consecutive years. 
The client inclusion criteria and the length of time clients will be in the project will be 
determined by a Project Advisory Committee that is being established with representation from 
relevant state agencies, providers, advocates, and consumers. 

Measures 

The VSTCP is comprised of four components, the Daily Adjustment Component (DAC), 
the Bi-annual Adjustment Component (BAC), the Service Tracking Component (STC), and the 
Consumer Satisfaction Component (CSC). Each component will be described separately. 



Daily Adjustment Component (DAO 

The DAC consists of a behavior checklist of adjustment indicators which is designed to 
measure those behaviors which place a child at risk of placement into a more restrictive 
setting. It is a daily checklist which requires the respondent to record on an optical 
scanner form whether a behavior did or did not occur. The checklist measures 
occurrence or non-occurrence as opposed to the frequency of occurrence. A backup 
respondent will be designated in the event that the primary respondent is unavailable. 
Social validity and inter-rater reliability are currently being assessed. Completion of the 
DAC should not require more than 1-3 minutes per child It is intended to be used 
wherever the child resides, whether in the home or in a substitute-care setting. See 
Appendix A for a list of each of the adjustment indicators and definitions. A training 
manual is currently being developed to help respondents learn critical components of 
each definition. 

Bi-annual Adjustment Component (BAC) 

The BAC is composed of the parent (CBCL), teacher (TRF), and youth (YSR) behavior 
checklists designed by Tom Achenbach and Craig Edelbrock. The CBCL will be 
administered to parents or residential staff every six months. The TRF will be 
administered to educational staff and the YSR will be administered to youth 1 1 years of 



age or older. They each relate to the childs' adjustment over a preceding 6 month 
period of time and focus on both social competence and problem behaviors, These 
measures have excellent reliability and validity. All of these checklists require 
approximately IS minutes to complete. 

Service Tracking Component (STQ 

The Service Tracking Component will be designed to provide weekly information 
regarding the provision of services to a child. It is of interest to determine where 
services are being provided and the nature and frequency of those services. Records will 
be kept of changes in residential or educational setting, and significant service delivery 
changes within either of those settings. This form is currently in development. 

Consumer Satisfaction Component (CSQ 

The Consumer Satisfaction Component (CSC) is comprised of Consumer Satisfaction 
Questionnaires that are being designed for parents, children, and adolescents. The child's 
parent(s) or legal guardian will fill out a Consumer Satisfaction Questionnaire even- six 
months. Children and adolescents who have given their assent will also be given a 
Consumer Satisfaction Questionnaire bi-annually. It is anticipated that these 
questionnaires will require approximately 15-30 minutes to complete. 

Consent 

Informed consent to place the child on the tracking system will be obtained from the 
child's parent or the Commissioner of SRS if the child is in the custody of the State. Informed 
consent will also be obtained from each individual respondent with respect to their participation, 
In those cases where staff rotation would preclude the designation of a single respondent, 
consent will be obtained from the program director. 

Those portions of the system for which the child or youth is asked to be the respondent 
(ie. the YSR and Child or Youth Consumer Satisfaction Questionnaire) will require their assent 
provided they are 11 years of age or older. For those children who do not wish to participate, 
data will be collected from adult respondents only. 

Timeline 

The Daily Adjustment Component of the tracking system is presently being field tested 
on a small number of clients who are receiving services within different components of the 
Vermont System of Care. In addition data relating to social validity and reliability is currently 
being collected. Undergraduate students will implement the system and are currently taking part 
in an extensive training process. It is anticipated that all components of the tracking system will 
in place by late February. 

Any questions regarding this project should be addressed to either John Burchard or 
Mark Schaefer in the Psychology Department at the University of Vermont. Phone: (802) 656- 
2670. 
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DAILY ADJUSTMENT INDICATORS 

Directions: Please indicate according to your best judgement whether or not the 

following behaviors or events have occurred on this day. Fill in the appropriate circle. (Y =YES, N=NO) 



ID#. 



Week beginning 



(MONTH/ DAY/YEAR) 



: 1. PHYSICS AGGRESSION: Did the child or youth hit, stiWbite, or scratch a person 
with intent to harm them? (Includes hitting with an object.) 




? V 1 
oo 


OO 


'• f'v'F 

iOO||^:;:: : 


oo 


s ^ 

O O 


2. 


.V. •.".■WW- ft ■ • ••• ■ Vfi 1 1TW1 .-..-•-.-» •O.IVI. •«>•-• -..*r ........ ......... . - . . ---- . 

PROPERTY DAMAGE: Via, the ciuia or youtn damage property on purpose i 
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THEFT: Did the child or youth take property without permission? 
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RUNAWAY: Did the child or youth run away? 
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ALCOHOL/DKUIj Uac: ma tire ciulu oryouui use arugs or aiconoi wiinoui pennissionr 
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SEXUAL ACTING OUT: Did the child or youth engage in inappropriate sexual behavior 
which was displayed publicly or directed toward another person? 
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EXTREME VERBAL ABUSE: Did the child or youth speak to another person in an extremely 
malicious, abusive or intimidating manner? 
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SAD: Was the child or youth sad, withdrawn, or depressed to a degree which significantly 
interfered with participation in an important activity? 
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AhMO^JS: Was the <Md or youth fearful, anxious, or worried to a degree which significantly 
interfered ivith participation in ar important activity? 
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10. SELF-INJURY: Did the child or youth harm or attempt to harm him or herself nonacddentally? 
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11. INAPPROPRIATE BOWEL MOVEMENTS: Did the child or youth intentionally smear 
his or her feces or deposit them in an inappropriate place? 
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12. F IFE THREAT: Did the child or youth threaten oi engage in physical assault in a manner which 
which you believe was life threatening? 
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13. SEXUAL ABUSE/ASSAULT: Did the child or youth attempt to force him or herself upon 
ERIC another person sexually? 
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DAILY ADJUSTMENT INDICATORS, SIDE 2 



14. SUICIDE ATTEMPT: Did the child or youth attempt to commit suicide? M T W T P S S 

©o ©o ©o oo ©o ©a ©e 

? > 15. FIRESETTINGiDidthe child or ybuth set a fire without permission or sett fire ^ ^ ■' ^ d ^ 
• 4 • 1ft a manner which could have resulted in property damage or harm to others? \ Wo 4 ^ B W^lfe'"' 

16. STRANGE BEHAVIOR: Did the youth have delusions, hallucinations, obsessions, compulsions, M T W T F S S 

or other bizarre behavior which significantly interfered with an important activity? ©o ©o ©O ©6 ©6 00 OO 

' ■ ■*•■">■ mmmmmmm mum mm ■:- 

> CRUELTY TO ANIMALS: Did the chili outh torture, kill or behave ^tieUy toward M T w T P s S 

:::: ^ - *# •• 

18. SELF-CONFIDENCE: Did the child or youth appear self-confident in his or her activities for M T W T F S S 

more than 85% of the time? ®e ®e $9 $4 00 00 09 

COMPLIANCE: Was the cnikfr or youm^ respond requests and general activity 3> : :v : § " : ; . 

iiiHP^- acceptable 85% of the time? < <• • ©O ©o OO OO ©O OO OO : 

20. PEER INTERACTIONS: Did the child or youth have good peer/sibling relations 
85% of the time? (Please check □ if no peer or sibling contact on this day.) 

i;;?^sch66 

for school attendance? (Please check Q if thfe was no school today.) 

22. PARENT CONTACT: Did the child or youth have contact with his or her natural or adoptive 
parentis) on this day? (Includes letter, telephone call, or personal visit.) 

! 23. VOCATIONAL INVOLVEMENTS Did the child or youm wo* for pay (check "P") or as 
IISS': * vol ^ tee l: f^? (If thechild or youth did not work check □) 

24. POLICE CONTACT: Did the child or youth have contact with the police concerning 

his or her negative or suspicious behavior? 

25. ^ 
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APPENDIX D 



SOME POSSIBLE MEASURES FOR IMPROVEMENTS TO THE CASSP 

SYSTEM AND SERVICE DELIVERY 



SOME POSSIBLE MEASURES OF IMPROVEMENTS 
TO THE CASSP SYSTEM AND TO ITS SERVICE DELIVERY 



Michael Hendricks, Ph.D. 

MH Associates 
3419 30th Street, N.W. 
Washington, DC 20008 

(202) 966-8066 



Presented at: 

CASSP Technical Assistance Research Meeting 
Georgetown University 
Washington, DC 



May 2, 1990 
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SIX (UNOFFICIAL) GOALS OF CASSP 



#1 s Increase the visibility and priority of children's 
mental health services 



§2i Increase funding for children's mental health 
services across the continuum of care 



#3: Serve more children as a ppropriately as possible 



Hit Improve the efficiency of service delivery among 
different child-serving agencies 



§'$: Involve families in planning and treatment efforts 



#6 : Assure that services are provided In a culturally 
sensitive manner 



ftp 




GOAL #1 : INCREASE THE VISIBILITY AND PRIORI TY OF ..CTILPREMlg 
MENTAL HEALTH SERVICES 

System Improvements? 

* Establish a focal point for children's mental health 
services 

* Develop top-level support for children's mental health 
services 

* Develop public support for children's mental health services 

* Develop a planning process that involves all key players 

* Define the target population 

* Complete a needs assessment of the target population 

* Develop a children's mental health plan 

* Include children and adolescents as a priority population in 
mental health plans or other official documents 

Service Delivery Improvements; 

w Mur i ii -i i r n i i ii i 1 1 ii ■ ii m i i biii ummmum m * m r Ti ■ mm i i m ■m ir rnr r 

* Identify all needy children in the target population 
* 

* 
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GOAL #2: INCREASE FUNDIN G F OR CHILDREN'S MENTAL HEALTH SERVICES 
ACROSS THE CONTINUUM OF CARE 



Sys tem Imp rovement a ; 

* Increase the overall resources available for children's 
mental health services 

* Expand Medicaid reimbursement for needed services 

* Access nev funds from private sources 

* Increase the proportion of total mental health funds 
earmarked for children's services 



Service D elivery Improvem ents; 

* Increase the proportion of identified target population 
served 

* 



91 



GOAL #3; SERVE MORE CHILDREN AS APPROPRIATELY AS PO SSIBLE 
System Improvements ; 

* Modify legislation, regulations, policies, standards, etc. 
as needed to develop a full range of services 

* Reallocate resources, especially from out-of-state and 
institutional-based services to in-state and community-based 
services 

* Support mini-demonstrations of components of continuum of 
care 

* stimulate the supply of trained professionals 
* 

* 

Service Delivery Improvements: 

* Increase the number of slots available in each core service 

* Decrease the total number, relative proportion, and length 
of restrictive placements {e.g*, out-of-state, out-of-home, 
institutional) 

* Increase the utilization of alternatives to hospitals and 
residential treatment centers 

* 
* 
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GOAL 04; IMPROVE THE EFFICIENCY OF SERVICE DELIVERY AMONG 
DIFFERENT CHILD-SERVING AGENCIES 

System Improvements; 

* Establish formal coordination mechanisms (committees, task 
forces , etc.) 

* Establish formal interagency agreements 

* Develop joint funding initiatives 

* Develop joint programming initiatives 

* Develop joint case-management policies and procedures 



Service Delivery Improvements; 

* Increase the total number and relative proportion of 
children with case managers 

* Establish an individual treatment plan for each child 
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GOAL #5; INVOLVE FAMILIES IN PLANNING AND TREATMENT EFFORTS 



System Improvements; 

* Develop new family support and/or advocacy groups (e.g., 
sponsor conferences or workshops, train family members, 
provide information) 

* Include families on key policymaking, planning, and advisory 
groups 

* Involve families in developing the State plan and other 
planning efforts 

* 



* 



Service Delivery Improvements; 

* Involve each child's family in treatment planning and 
service delivery 

* 



* 
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G OAL 06: ASSURE THAT SERVICES ARE PROVIDED IN A CULTURALLY 
SENSITIVE MANNER 



System Improvements: 

* Include minorities on key policymaking, planning, and 
advisory groups 

* Involve minorities in developing the State plan and other 
planning efforts 

* Conduct a special assessment of minority needs 

* Develop special plans for providing culturally sensitive 
services 

* Train staff regarding cultural sensitivity 

* Sponsor conferences, workshops, forums, etc. on cultural 
sensitivity 

Service Delivery Improvements! 

* Hire bilingual and/or bicultural staff 

* Develop specialized programs or services for minority 
populations 

* 
♦ 
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APPENDIX E 
PROTECTION OF HUMAN SUBJECTS 
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©PRR Reports 

■ NIH PHS HHS 



PROTECTION 
OF 

HUMAN SUBJECTS 



CODE OF FEDERAL REGULATIONS 

45 CFR 46 



Revised as of March 8, 1983 
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NATIONAL RESEARCH ACT 
PUBLIC LAW 93-348 
JULY 12, 1974 

INSTITUTIONAL REVIEW BOARDS; ETHICS GUIDANCE PROGRAM 

Sec. 212. (a) Part I of title IV of the Public Health Service Act, as 
amended by section 103 of this Act, is amended by adding at the end the 
following new section: 

M INSTITUTIONAL REVIEW BOARDS; ETHICS GUIDANCE PROGRAM 

"Sec, 474. (a) The Secretary shall by regulation require that each entity 
which applies for a grant or contract under this Act for any project or 
program which involves the conduct of biomedical or behavioral research 
involving human subjects submit in or with its application for such grant or 
contract assurances satisfactory to the Secretary that it has established (in 
-accordance with regulations which the Secretary shall prescribe) a board (to 
be known as an Institutional Review Board 9 ) to review biomedical and 
behavioral research involving human subjects conducted at or sponsored by 
such entity in order to protect the rights of the human subjects of such 
research. 

"(b) The Secretary shall establish a program within the Department 
under which requests for clarification and guidance with respect to ethical 
issues raised in connection with biomedical or behavioral research involving 
human subjects are responded to promptly and appropriately/ 9 

(b) The Secretary of Health, Education, and Welfare shall within 240 
days of the date of the enactment of this Act promulgate such regulations as 
may be required to carry out section 474(a) of the Public Health Service 
Act. Such regulations shall apply with respect to applications for grants and 
contracts under such Act submitted after promulgation of such regulations. 

THE CODE OF FEDERAL REGULATIONS, 

45 CFR 46, IMPLEMENTS THESE AMENDMENTS 

TO THE PUBLIC HEALTH SERVICE ACT. 



salus ubi multi comiliarii 
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CODE OF FEDERAL REGULATIONS 



TITLE 45 
PUBLIC WELFARE 

DEPARTMENT OF HEALTH AND HUMAN SERVICES 

NATIONAL INSTITUTES OF HEALTH 
OFFICE FOR PROTECTION FROM RESEARCH RISKS 




PART 46— PROTECTION OF HUMAN SUBJECTS 
REVISED AS OF MARCH 8, 1983 
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4$ CFR 46 



PART 46—PROTECTION OF 
HUMAN 
SUBJECTS 

Subpart A—Basic HHS Policy for 
Protection of Human Research 
Subjects 

Sec. 

46.101 To what do these regulations apply? 

46.102 Definitions. 

46.103 Assurances. 

46. 104 Section reserved. 

46. 105 Section reserved . 

46.106 Section reserved. 
4u 107 IRB membership. 

46. 108 IRB functions and operations. 

46. 109 IRB review of research. 

46.110 Expedited review procedures for 
certain kinds of research involving no 
more than minimal risk, and for minor 
changes in approved research. 

46. 1 1 1 Criteria for IRB approval of 
research. 

46.112 Review by institution. 

46. 1 13 Suspension or termination of IRB 
approval of research. 

46. 1 14 Cooperative research. 

46.115 IRB records. 

46.1 16 General requirements for informed 
consent. 

46. 1 17 Documentation cf informed 
consent. 

46. 1 18 Applications and proposals lacking 
definite plans for involvement of human 
subjects. 

46. 1 19 Research undertaken without the 
intention of involving human subjects. 

46.120 Evaluation and disposition of 
applications and proposals. 

46.121 Investigational new drug or device 
30-day delay requirement. 

46.122 Use of federal funds. 

46. 1 23 Early termination of research 
funding: evaluation of subsequent 
applications and proposals. 

46.124 Conditions. 

Subpart B— Additional Protections 
Pertaining to Research f 
Development, and Related 
Activities Involving Fetuses, 
Pregnant Women, and Human 
In Vitro Fertilization 

Sec. 

46.201 Applicability. 

46.202 Purpose. 

46.203 Definitions. 

46.204 Ethical Advisory Boards. 

46.205 Additional duties of the Institutional 
Review Boards in connection with 
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activities involving fetuses, pregnant 
women, or human in vitro fertilization. 

46.206 Genera) limitations, 

46.207 Activities directed toward pregnant 
women as subjects. 

4*6.208 Activities directed toward fetuses in 
utero hi subjects. 

46.209 Activities directed toward fetuses ex 
utero, including nonviable fetuses, as 
subjects. 

46.210 Activities involving the d«ad fetus, 
fetal material, or the placenta. 

46.21 1 Modification or waiver «>f specific 
requirements. 

Subpart C— Additional Protections 
Pertaining to Biomedical and 
Behavioral Research Involving 
Prisoners as Subjects 

Sec. 

46.301 Applicability. 

46.302 Purpose. 

46.303 Definitions. 

46.304 Composition of institutional Review 
Boards where ptisoners are involved. 

46.305 Additional duties of the Institutional 
Review Boards where prisoners are 
involved. 

46.306 Permitted activities involving 
prisoners. 



Subpart D— Additional Protections 
for Children Involved as Subjects in 
Research 



Sec. 

46.401 To what do these regulations apply? 

46.402 Definitions. 

46.403 IRB duties. 

46.404 Research not involving greater than 
minimal risk. 

46.403 Research involving greater than 
minimal risk but presenting the prospect 
of direct benefit to the individual subjects. 

46.406 Research involving greater than 
minimal risk and no prospect of direct 
benefit to individual subjects, but Xikely to 
yield generalizable knowledge about the 
subject's disorder or condition. 

46.407 Research not otherwise approvable 
which presents an opportunity to 
understand, prevent, or alleviate a serious 
problem affecting tl|e health or welfare of 
children. 

46.408 Requirements for permission by 
parents or guardians and for assent by 
children. 

46.409 Wards. 

Authority: 5 U S C. 301; sec. 474(a), 88 
Stat. 352 (42 U.S.C. 289/-3<a)>. 
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Subpart A — Basic HHS PoKcy for 
Protection of Human Research 
Subjects 

Source: 46 FR 838ft. January 26. 198*, 48 FR 
9269, March 4. 1983. 

5 46.101 To what do these 
regulations apply? 

(a) Except as provided in 
paragraph (b) of this .section, this 
subpart applies to all research 
involving human subjects conducted 
by the Department of Health and 
Human Services or funded in whole 
or in part by a Department grant, 
contract, cooperative agreement or 
fellowship. 

(1) This includes research 
conducted by Department employees, 
except each Principal Operating 
Component head may adopt such 
nonsubstantive, procedural 
modifications as may be appropriate 
from an administrative standpoint. 

(2) It also includes research 
conducted or funded by the 
Department of Health and Human 
Services outside the United States, 
but in appropriate circumstances, the 
Secretary may, under paragraph (c) of 
shis section waive the applicability of 
some or all of the requirements of 
these regulations for research of this 
type. 

(b) Research activities in which the 
only involvement of human subjects 
will be in one or more of the 
following categories are exempt from 
these regulations unless the research 
is covered by other subparts of this 
part: 

(1) Research conducted in 
established pr commonly accepted 
educational settings, involving 
normal educational practices, such as 

(i) research on regular and special 
education instructional strategics, or 

(ii) research on the effectiveness of or 
the comparison among instructional 
techniques, curricula, or classroom 
management methods. 

(2) Research involving the use of 
educational tests (cognitive, 
diagnostic, aptitude, achievement), if 
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information taken from these sources 
t$ recorded in such a manner that 
subject* cwnot be identified, directly 
or through identifiers linked to the 
subjects. 

(3) Research involving survey or 
interview procedures, except where 
all of the following conditions exist: 
(s) responses are recorded in such a 
manner that the human subjects can 
be identified , directly or through 
identifiers linked to the subjects, (it) 
the subject's responses, if they 
became known outside the vesearch, 
could reasonably place the subject at 
risk of criminal or civil liability or be 
damaging to the subject's financial 
standing or employability, and (iii) 
the research deals wiui sensitive 
aspects of the subject's own behavior, 
such as illegal conduct, drug use, 
sexual behavior* or use of alcohol. 
All research involving survey or 
interview procedures is exempt, 
without exception, when the 
respondents are elected 0; appointed 
public officials or candidates for 
public office. 

(4) Research involving the 
observation (including observation by 
participants) of public behavior, 
except where all of the following 
conditions exist: (i) observations are 
recorded in such a manner that the 
human subjects can be identified, 
directly or through identifiers linked 
to the subjects, (ii) th& observations 
recorded about the individual, if they 
became known outside the research, 
could reasonably place the subject at 
risk of criminal or civil liability or be 
damaging to the subject's financial 
standing or employability, and (iii) 
the research deals with sensitive 
aspects of the subject's own behavior 
such a« Itegal conduct, drug use, 
sexual behavior, or use of alcohol. 

(5) Research involving the 
collection or study of existing data, 
documents* records, pathological 
specimens, or diagnostic specimens, 
if these sources are publicly available 
or if the information is recorded by 
the investigator in such a manner that 



subjects cannot be identified, directly 
or through identifiers linked to the 
subjects. 

(6) Unless specifically required by 
statute (and except to the extent 
specified in paragraph (t)), research 
and demonstration projects which 
are conducted by or subject to the 
approval of the Department of 
Health and Human St* vices, an J 
which are designed to study, 
evaluate, or otherwise examine; (i) 
programs under the Social Security 
Act, or other public benefit or 
service programs; (ii) procedures for 
obtaining benefits or services under 
those programs; (iii) possible changes 
in or alternatives to those programs 
or procedures; or (iv) possible 
changes in methods or levels of 
payment for benefits or services 
under those programs. 

(c) The Secretary has final 
authority to determine whether a 
particular activity is covered by these 
regulations. 

(d) The Secretary may require that 
specific research activities or classes 
of research activities conducted or 
funded by the Department* but not 
otherwise covered by these 
regulations, comply with some or all 
of these regulations. 

(e) The Secretary may also waive 
applicability of these regulations to 
specific research activities or ctasses 
of research activities, otherwise 
covered by these regulations. Notices 
of these actions will be published in 
thn Federal Register as they occur* 

(0 No individual may receive 

Department funding for research 

covered by these regulations unless 

the individual is affiliated with or 

sponsored by an institution which 

assumes responsibility for the 

research under an assurance satisfying 
the requirements of this part, or the 

individual makes other arrangements 

with the Department. 

(g) Compliance with these 

regulations will in no way render 

inapplicable pertinent federal, state, 

or local taws or regulations. 
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(h) Each subpart of these 
regulations contains a separate 
section describing to what the subpart 
applies. Research which is covered 
by more than one subpart shall 
comply with u\\ applicable subparts. 

(i) If, following review of 
proposed research activities that are 
exempr f;om these regulations under 
paragraph (b)(6), the Secretary 
determines that a research or 
demonstration project presents & 
danger to the physical, mental, or 
emotional well-being of a participant 
or subject of the research or 
demonstration project, then federal 
funds may not be expended for such 
a project without the written, 
informed consent of each participant 
or subject. 

§ 46.102 Definitions. 

(a) "Secretary" means the 
Secretary of Health and Human 
Services and any other officer or 
employee of the Department of 
Health and Human Services to whom 
authority has been delegated. 

(b) "Department" or "HHS" 
means the Department of Health and 
Human Services. 

(c) "Institution" means any public 
or private entity or Agency (including 
federal, state, and other agencies). 

<d) "Legally authorised 
representative" means an individual 
or judicial 01 other body authorised 
under applicable law to consent on 
behalf of a prospective subject to the 
subject's participation in the 
procedure(s) involved in the research. 

(e) "Research 1 * means a 
systematic investigation designed to 
develop or contribute to generalizable 
knowledge. Activities which meet 
this definition constitute "research" 
for purposes of these regulations, 
whether or not they are supported or 
funded under a program which is 
considered research for other 
purposes. For example, some 
"demonstration 4 * and "service" 
programs may include research 
activities. 
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(0 "Human subject mtans a 
living individual about whom an 
investigator (whether professional or 
student) conducting research obtains 
(I) data through intervention or 
interaction with the individual, 01 (2) 
identifiable private information. 
9 Intervention" includes both 
physical prwedures by which data are 
gathered (for example, venipuncture) 
and manipulations of the subject or 
the subject's environment that are 
performed for research purposes. 
"Interaction" includes 
communication or interpersonal 
contact between investigator and 
subject. 4 'Private information" 
includes information about behavior 
that occui i in a context in which an 
individual can reasonably expect that 
no observation or recording is taking 
place, and information which has 
been provided for specific purposes 
by an individual arid which the 
individual can reasonably expect will 
not be made public (for example, a 
medical record). Private information 
must be indjvidu&llyHdemifiabIc 
(i.e., the identity of the subject is or 
may readily be ascertained by the 
investigator or associated with the 
information) in order for obtaining 
the information to constitute research 
involving human subjects. 

(g) "Minimal risk" means that the 
risks of harm anticipated in the 
proposed research ?*e not greater, 
considering probability and 
magnitude, than those ordinarily 
encountered in daily life or during the 
performance of routine physical or 
psychological examinations or tests. 

(h) "Certification" means the 
official notification by the institution 
to the Department in accordance with 
the requirements of this part that a 
research project or activity involving 
human subjects has been reviewed 
and approved by the Institutional 
Review Board (IRB) in accordance 

v. S the approved assurance on file at 
HHS. (Certification is required when 
the research is funded by the 
Department and not otherwise exempt 
in accordance with § 46.101(b)). 



t 46.103 Assurances* 

(a) Each institution engaged in 
research covered by these regulations 
shall provide written assurance 
satisfactory to the Secretary that it 
will comply with the requirements set 
forth in these regulations. 

(b) The Department wil! conduct or 
fund research covered by these 
regulations only if the institution has 
an assurance approved as provided in 
this section, and crny if the institution 
has certified to the Secretary (hat the 
research has been reviewed and 
approved by an IRB provided for in 
the a»ur *e< and will be subject \o 
continuing review by the IRB. This 
assurance shall at a minimum include: 

(DA statement or principles 
governing the institution in the 
discharge of its responsibilities for 
protecting the rights and welfare of 
human subjects of research conducted 
at or sponsored by the institution, 
regardless of source of funding. This 
may include an appropriate existing 
code, declaration, or statement of 
ethical principles, or a statement 
formulated by the institution itself. 
This requirement does not preempt 
provisions of these regulations 
applicable to Department-funded 
research and is not applicable to any 
research in an exempt category listed 
in § 46.101. 

(2) Designation of one or more 
IRBs established in accordance with 
the requirements of this subpart, and 
for which provisions are made for 
meeting space and sufficient staff to 
support the IRB's review and 
recordkeeping duties. 

(3) A list of the IRB members 
identified by name; earned degrees; 
representative capacity; indications of 
experience such as board 
certifications, licenses, etc., 
sufficient to describe each member's 
chief anticipated contributions to IRB 
deliberations; and any employment or 
other relationship between each 
member and the institution; for 
example: full-time employee, part- 
time employee* member of governing 
panel or board* stockholder* paid or 



unpaid consultant. Changes in (RB 
membership shall be reported to the 
Secretary. 1 

(4) Written procedures which the 
IRB will follow (i) fo? wOflducting its 
initial and continuing review of 
research and for reporting its findings 
and actions to the investigator and the 
institution; (ii) for dctermwng which 
projects require re new more often 
than annually and which projects 
need verification from somtes other 
than the investigators that no material 
change; have occurred since previous 
IRB review; (iii) for insuring prompt 
reporting to the IRB of proposed 
changes in a research activity, and for 
insuring that changes in approved 
research, during the period for which 
IRB approval has already been given* 
may not be initiated withoot IRB 
review and approval except where 
necessary to eliminate apparent 
immediate hazards to the subject; and 
(iv) for insuring prompt reporting to 
the IRB and to the Secretary 1 of 
unanticipated problems involving 
risks to subjects or others. 

(c) The assurance shall be executed 
by an individual authorized to act for 
the institution and to assume on 
behalf of the institution the 
obligations imposed by these 
regulations, and shall be filed in such 
form and manner as the Secretary 
may prescribe. 

(d) The Secretary will evaluate all 
assurances submitted in accordance 
with these regulations through such 
officers and employees of the 
Department and such experts or 
consultants engaged for this purpose 
as the Secretary determines to be 
appropriate. The Secretary's 
evaluation, will take into 
consideration the adequacy of the 
proposed IRB in light of the 
anticipated scope of the institution's 
research activities and the types of 
subject populations likely to be 

1 Reports should be filed with the Office 
for Protection from Research Risks* National 
Institutes of Health, Department of Health 
and Human Services. Bethesda. Maryland 
20205. 
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involved, the appropriateness of the 
proposed initial and continuing 
review procedures in light of the 
probable risks, and the size and 
complexity of the institution. 

(e) Gil the basis of this evaluation, 
the Secretary may approve or 
disapprove the assurance , or enter 
into negotiations to develop an 
approvable one. The Secretary may, 
limit the period during which any 
particular approved assurance or class 
of approved assurances shall remain 
effective or otherwise condition or 
restrict approval . 

(0 Within 60 days after the date of 
submission to HHS of an application 
or proposal, an institution with an 
approved assurance covering the 
proposed research shall certify that 
the application or proposal has been 
reviewed and approved by the IRB. 
Other institutions shall certify that the 
application or proposal has been 
approved by the IRB within 30 days 
after receipt of a request for such a 
certification from the Department. If 
the certification is not submitted 
within these time limits, the 
application or proposal may be 
returned to the institution. 

i 46.104 [Reserved] 

5 46.105 [Reserved] 

S 46.106 [Reserved] 

5 46.107 IRB membership. 

(a) Each IRB shall have at least 
five members, with varying 
backgrounds to promote complete and 
adequate review of research activities 
commonly conducted by the 
institution The IRB shall be 
sufficiently qualified through the 
experience and expertise of its 
members, and the diversity of the 
members* backgrounds including 
consideration of the racial and 
cultural backgrounds of members and 
sensitivity to such issues as 
community attitudes, to promote 
respect for its advice and counsel in 
safeguarding the rights and welfare of 
human subjects. In addition to 



possessing the professional 
competence necessary to review 
specific research activities, the IRB 
shall be able to ascertain the 
acceptability of proposed research in 
terms of institutional commitments 
and regulations, applicable law, and 
standards of professional conduct and 
practice. The IRB shall therefore 
include persons knowledgeable in 
these areas. If an IRB regularly 
reviews research that involves a 
vulnerable category of subjects, 
including but not limited to subjects 
covered by other subparts of this part, 
the IRB shall include one or more 
individuals who are primarily 
concerned with the welfare of these 
subjects. 

(b) No IRB may consist entirely of 
men or entirely of women, or entirely 
of members of one profession. 

(c) Each IRB shall include at least 
one member whose primary concerns 
are in nonscientific areas; for 
example: lawyers, ethicists, members 
of the clergy. 

(d) Each IRB shall include at least 
one member who is not otherwise 
affiliated with the institution and who 
is not part of the immediate family of 
a person who is affiliated with the 
institution. 

(e) No IRB may have a member 
participating in the IRB*s initial or 
continuing review of any project in 
which the member has a conflicting 
interest, except to provide 
information requested by the IRB. 

(0 An IRFi may, in its discretion, 
invite individuals with competence in 
special areas to assist in the review of 
complex issues which require 
expertise, beyond or in addition to that 
available on the IRB. These 
individuals may not vote with the 
IRB. 

9 46.108 IRB functions and 
operations. 

In order to fulfill the requirements 
of these regulations each IRB shall: 

(a) Follow written procedures as 
provided in § 46.103(b)(4), 



(b) Except when an expedited 
review procedure is used (see 

§ 46. 1 10), review proposed research 
at convened meetings at which a 
majority of the members of the IRB 
are present, including at least one 
member whose primary concerns are 
in nonscientific areas. In order for the 
research to be approved, it shall 
receive the approval of a majority of 
those members present at the 
meeting. 

(c) Be responsible for reporting to 
the appropriate institutional officials 
and the Secretary 1 any serious or 
continuing noncompliance by 
investigators with the requirements 
and determinations of the IRB. 

§ 46.109 IRB review of research. 

(a) An IRB shall review and have 
authority to approve, require 
modifications in (to secure approval)*, 
or disapprove all research activities 
covered by these regulations. 

(b) An IRB shall require that 
information given to subjects as part 
of informed consent is in accordance 
with § 46.116. The IRB may require 
that information, in addition to that 
specifically mentioned in § 46. 1 16, 
be given to the subjects when in the 
IRB's judgment the information 
would meaningfully add to the 
protection of the rights and welfare of 
subjects. 

(c) An IRB shall require 
documentation of informed consent or 
may waive documentation in 
accordance with § 46. 117. 

(d) An IRB shall notify 
investigators and the institution in 
writing of its decision to approve or 
disapprove the proposed research 
activity, or of modifications required 
to secure IRB approval of the 
research activity. If the IRB decides 
to disapprove a research activity, it 
shall include in its written notification 



1 Reports should be filed with the Office 
for Protection from Research Risks, National 
Institutes of Health, Department of'Hcalth 
and Human Services, Bethesda, Maryland 

20205. 
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a statement of the reasons for its 
decision and give the investigator an 
opportunity to respond in person or in 
writing. 

(e) An IRB shall conduct 
continuing review of research covered 
by these regulations at intervals 
appropriate to the degree of risk, but 
not less than once per year, and shall 
have authority to observe or have a 
third party obseive the consent 
process and the research. 

(46.110 Expedited review 
procedures for certain kinds of 
research involving no more than 
minimal risk, and for minor 
changes in approved research. 

(a) The Secretary has established, 
and published in the Federal 
Register, a list of categories of 
research that may be reviewed by the 
IRB through an expedited review 
procedure. The list will be amended, 
as appropriate, through periodic 
republication in the Federal 
Register. 

(b) An IRB may review some or all 
of the research appearing on the list 
through an expedited review 
procedure, if the research involves no 
more than minimal risk. The IRB may 
also use the expedited review 
procedure to review minor changes in 
previously approved research during 
the period for which approval is 
authorized. Under an expedited 
review procedure, the review may be 
carried out by the IRB chairperson or 
by one or more experienced reviewers 
designated by the chairperson from 
among members of the IRB. In 
reviewing the research, the reviewers 
may exercise all of the authorities of 
the IRB except that the reviewers may 
not disapprove the research. A 
research activity may be disapproved 
only after review in accordance with 
the non-expedited procedure set forth 
in 8 46.108(b). 

(c) Each IRB which uses an 
expedited review procedure shall 
adopt a method for keeping all 
members advised of research 



proposals which have been approved 
under the procedure. 

(d) The Secretary may restrict, 
suspend* or terminate an institution's 
or IRB's use of the expedited review 
procedure when necessary to protect 
the rights or welfare of subjects. 

846.111 Criteria for IRB 
approval of research . 

(a) In order to approve research 
covered by these regulations the IRB 
shall determine that all of the 
following requirements are satisfied: 

(1) Risks to subjects are 
minimized: (i) By using procedures 
which are consistent with sound 
research design and which do not 
unnecessarily expose subjects to risk, 
and (ii) whenever appropriate, by 
using procedures already being 
performed on the subjects for 
diagnostic or treatment purposes. 

(2) Risks to subjects are reasonable 
in relation to anticipated benefits, if 
any, to subjects, and the importance 
of the knowledge that may reasonably 
be expected to result. In evaluating 
risks and benefits, th: IRB should 
consider only those risks and benefits 
that may result from the research (as 
distinguished from risks and benefits 
of therapies subjects would receive 
even if not participating in the 
research). The IRB should not 
consider possible long-range effects 
of applying knowledge gained in the 
research (for example, the possible 
effects of the research on public 
policy) as among those research risks 
that fall within the purview of its 
responsibility. 

(3) Selection of subjects is 
equitable. In making this assessment 
the IRB should take into account the 
purposes of the research and the 
setting in which the research will be 
conducted. 

(4) Informed consent Will be 
sought from each prospective subject 
or the subject's legally authorized 
representative, in accordance with, 
and to the extent required by 

§ 46.116. 

U-i 



(5) Informed consent will be 
appropriately documented, in 
accordance with, and to the extent 
required by § 46.117. 

(6) Where appropriate, the research 
plan makes adequate provision for 
monitoring the data collected to 
insure the safety of subjects. 

(7) Where appropriate, there are 
adequate provisions to protect the 
privacy of subjects and to maintain 
the confidentiality of data. 

(b) Where some or all of the 
subjects are likely to be vulnerable to 
coercion or undue influence, such as 
persons with acute or severe physical 
or mental illness, or persons who are 
economically or educationally 
disadvantaged, appropriate additional 
safeguards have been included in the 
study to protect the rights and welfare 
of these subjects. 

8 46. 1 12 Review by institution. 

Research covered by these 
regulations that has been approved by 
an IRB may be subject to further 
appropriate review and approval or 
disapproval by officials of the 
institution. However, those officials 
may not approve the research if it has 
not been approved by an IRB. 

t 46, 1 13 Suspension or 
termination of IRB approval of 
research. 

An IRB shall have authority to 
suspend or terminate approval of 
research that is not being conducted 
in accordance with the IRB's 
requirements or that has been 
associated with unexpected serious 
harm to subjects. Any suspension or 
termination of approval shall include 
a statement of the reasons for the 
IRB's action and shall be reported 
promptly to the investigator, 
appropriate institutional officials, and 
the Secretary. 1 



1 Reports should be filled with (he Office 
for Protection from Research Risks, National 
Institutes of Health, Department of Health 
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I 46.114 Cooperative research. 

Cooperative research projects are 
those projects, normally supported 
through grants, contracts, or similar 
arrangements, which involve 
institutions in addition to the grantee 
or prime contractor (such as a 
contractor with the grantee, or a 
subcontractor with the prime 
contractor). In such instances, the 
grantee or prime contractor remains 
responsible to the Department for 
safeguarding the rights and welfare of 
human subjects. Also, when 
cooperatirg institutions conduct some 
or all of the research involving some 
or all of these subjects, each 
cooperating institution shall comply 
with these regulations as though it 
received funds for its participation in 
the project directly from the 
Department, except that in complying 
with these regulations institutions 
may use joint review, reliance upon 
the review of another qualified IRB, 
or similar arrangements aimed at 
avoidance of duplication of effort. 

i 46.115 IRB records. 

(a) An institution, or where 
appropriate an IRB, shall prepare and 
maintain adequate documentation of 
IRB activities, including the 
following: 

(1) Cop ; 's of all research proposals 
reviewed, scientific evaluations, if 
any, that accompany the proposals, 
approved sample consent documents, 
progress reports submitted by 
investigators t and reports of injuries 
to subjects. 

(2) Minutes of IRB meetings which 
shall be in sufficient detail to show 
attendance at the meetings; actions 
taken by the IRB; the vote on these 
actions including the number of 
members voting for, against, and 
abstaining; the basis for requiring 
changes in or disapproving research; 
and a written summary of the 
discussion of controverted issues and 
their resolution. 

(3) Records of continuing review 
activities. 



(4) Copies of all correspondence 
between the IRB and the 
investigators. 

(5) A list of IRB members as 
required by § 46.103(b)(3). 

(6) Written procedures for the IRB 
as required by § 46.103(b)(4). 

(7) Statements of significant new 
findings provided to subjects, as 
required by § 46.116(b)(5). 

(b) The records required by this 
regulation shall be retained for at 
least 3 years after completion of the 
research, and the records shall be 
accessible for inspection and copying 
by authorized representative* of the 
Department at reasonable times and 
in a reasonable manner. 

{46.116 General requirements 
for informed consent. 

Except as provided elsewhere in 
this or other subparts, no investigator 
may involve a human being as a 
subject in research covered by these 
regulations unless the investigator has 
obtained the legally effective 
informed consent of the subject or the 
subject's legally authorized 
representative. An investigator shall 
seek such consent only under 
circumstances that provide the 
prospective subject or the 
representative sufficient opportunity 
to consider whether or not to 
participate and that minimize the 
possibility of coercion or undue 
influence. The information that is 
given to the subject or the 
representative shall be in language 
understandable to the subject or the 
representative. No informed consent, 
whether oral or written, may include 
any exculpatory language through 
which the subjec t or the 
representative is made to waive or 
appear to waive any of the subject's 
legal rights, or releases or appears to 
release the investigator, the sponsor, 
the institution or its agents from 
liability for negligence. 

(a) Basic elements of informed 
consent. Except as provided in 
paragraph (c) or (d) of this section, in 



seeking informed consent the 
following information shall be 
provided to each subject: 

(1) A statement that the study 
involves research, an explanation of 
the purposes of the research and the 
expected duration of the subject's 
participation, a description of the 
procedures to be followed, and 
identification of any procedures 
which are experimental; 

(2) A description of any reasonably 
foreseeable risks or discomforts to the 
subject; 

(3) A description of any benefits to 
the subject or to others which may 
reasonably be expected from the 
research; 

(4) A disclosure of appropriate 
alternative procedures or courses of 
treatment, if any, that might be 
advantageous to the subject; 

(5) A statement describing the 
extent, if any, to which 
confidentiality of records identifying 
the subject will be maintained; 

(6) For research involving more 
than minimal risk, an explanation as 
to whether any compensation and an 
explanation as to whether any 
medical treatments are available if 
injury occurs and, if so, what they 
consist of, or where further 
information may be obtained; 

(7) An explanation of whom to 
contact for answers to pertinent 
questions about the research "and 
research subjects' rights, and whom 
to contact in the event of a research* 
related injury to the subject; and 

(8) A statement that participation is 
voluntary, refusal to participate will 
involve no penalty or loss of benefits 
to which the subject is otherwise 
entitled, and the subject may 
discontinue participation at any time 
without penalty or loss of benefits to 
which the subject is otherwise 
entitled. 

(b) Additional elements of 
informed consent. When appropriate, 
one or more of the following elements 
of information shall also be provided 
to each subject: 
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(1) A statement that the particular 
treatment or procedure may involve 
risks to the subject (or to the embryo 
or fetus, if the subject is or may 
become pregnant) which are currently 
unforeseeable; 

(2) Anticipated circumstance^ 
under which the subject s 
participation may be terminated by 
the investigator without regard to the 
subject's consent; 

(3) Any additional costs to the 
subject that may result from 
participation in the research; 

(4) The consequences of a 
subject's decision to withdraw from 
the research and procedures for 
orderly termination of participation 
by the subject; 

(5) A statement that significant 
new findings developed during the 
course of the research which may 
relate to the subject's willingness to 
continue participation will be 
provided to the subject; and 

(6) The approximate number of 
subjects involved in the study. 

(c) An IRB may approve a consent 
procedure which does not include, or 
which alters, some or all of the 
elements of informed consent set 
forth above, or waive the requirement 
to obtain informed consent provided 
the IRB finds and documents that: 

(1) The research or demonstration 
project is to be conducted by or 
subject to the approval of state or 
local government officials and is 
designed to study, evaluate, or 
otherwise examine: (i) programs 
under the Social Security Act, o 
other public benefit or service 
programs; (ti) procedures for 
obtaining benefits or services under 
those programs; (iii) possible changes 
in or alternatives to those programs 
or procedures; or (iv) possible 
change; in methods or levels of 
payment for benefits or services under 
those programs; and 

(2) The research could not 
practicably be. carried out without the 
waiver or alteration. 

(d) An IRB may approve a consent 
procedure which does not include, or 



which alters, some or all of the 
elements 'of informed consent set 
forth above, or waive the 
requirements to obtain informed 
consent provided the IRB finds and 
documents that: 

(1) The research involves no more 
than minimal risk to the subjects; 

(2) The waiver or alteration will 
not adversely affect the rights and 
welfare of the subjects; 

(3) The research could not 
practicably be carried out without the 
waiver or alteration; and 

(4) Whenever appropriate, the 
subjects will be provided with 
additional pertinent information after 
participation. 

(e) The informed consent 
requirements in these regulations are 
not intended to preempt any 
applicable federal, state, or local laws 
which require additional information 
to be disclosed in order for informed 
consent to be legally effective. 

(0 Nothing in these regulations is 
intended to limit the authority of a 
physician to provide emergency 
medical care, to the extent the 
physician is permitted to do so under 
applicable federal, state, or local law. 

8 46. 1 17 Documentation of 
informed consent. 

(a) Except as provided in 
paragraph (c) of this section, 
informed consent shall be 
documented by the use of a written 
consent form approved by the IRB 
and signed by the subject or the 
subject's legally authorized 
representative. A copy shall be given 
to the person signing tne form. 

(b) Except as provided in 
paragraph (c) of this section, the 
consent form may be either of the 
following: 

(1) A written consent document 
that embodies the elements of 
informed consent required by 
S 46. 1 16. This form may be read to 
the subject or the subject's legally 
authorized representative, but in any 
event, the investigator shall give 
either the subject or the representative 



adequate opportunity to read it before 
it is signed; or 

(2) A "short form" written 
consent document stating thai the 
elements of informed consent 
required by § 46. 1 lo have been 
presented orally to. the subject or the 
subject's legally authorized 
representative. When this method is 
used, there shall be a witness to the 
oral presentation. Also, the IRB shall 
approve a written summary of what is 
to be said to the subject or the 
representative. Only the short form 
itself is to be signed by the subject or 
the representative. However, the 
witness shall sign both the short form 
and a copy of the nummary, and the 
person actually obtaining consent 
shall sign a copy of the summary. A 
copy of the summary shall be given to 
the subject or the representative, in 
addition to a copy of the "short 

form." 
(c) An IRB may waive the 

requirement for the investigator to 

obtain a signed consent form for some 

or all subjects if it finds either: 

( 1) That the only record linking the 
subject and the research would be the 
consent document and the ' ncipal 
risk would be potential hari.« resulting 
from a breach of confidentiality. Each 
subject will be asked whether the 
subject wants documentation linking 
the subject with the research, and the 
subject's wishes will govern; or 

(2) That the research presents no 
more than minimal risk of harm to 
subjects and involves no procedures 
for which written consent is normally 
required outside of the research 
context. 

In cases where jhe documentation 
requirement is waived, the IRB may 
require the investigator to provide 
subjects with a written statement 
regarding the research. 

9 46.118 Applications and 
proposals lacking definite plans for 
involvement of human subjects. 

Certain types of applications for 
grants, cooperative agreements, or 
contracts are submitted to the 
Department with the knowledge that 
subjects may be involved within the 
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period of funding, but definite plans 
would not normally be set forth in the 
application or proposal. These 
include activities such as institutional 
type grants (including bloc grants) 
where selection of specific projects is 
the institution's responsibility; 
research training grants where the 
activities involving subjects remain to 
be selected; and projects in which 
human subjects* involvement will 
depend upon completion of 
instruments, prior animal studies, or 
purification of compounds. These 
applications need not be reviewed by 
an 1RB before an award may be 
made. However, except for research 
described in § 46.101(b), no human 
subjects may be involved in any 
project supported by these awards 
until the project has been reviewed 
and approved by the IRB, as provided 
in these regulations, and certification 
submitted to the Department. 

i 46.119 Research undertaken 
without the Intention of involving 
human subjects. 

In the event research (conducted or 
funded by the Department) is 
undertaken without the intention of 
involving human subjects, but it is 
later proposed to use human subjects 
in the research, the research shall first 
be reviewed and approved by an IRB, 
as provided in these regulations, a 
certification submitted to the 
Department, and final approval given 
to the proposed change by the 
Department. 

§ 46.120 Evaluation and 
disposition of applications and 
proposals. 

(a) The Secretary will evaluate all 
applications and proposals involving 
human - lbjects submitted to the 
Department through such officers and 
employees of the Department and 
such experts and consultants as the 
Secretary determines to be 
appropriate. This evaluation will take 
into consideration the risks to the 
subjects, the adequacy of protection 
against these risks, the potential 
benefits of the proposed research to 



the subjects and others, and the 
importance of the knowledge to be 
gained. 

(b) On the basis of this evaluation, 
the Secretary may approve or 
disapp' ve the application or 
proposal, or enter into negotiations to 
develop an approvable one. 

5 46,121 Investigational new drug 
or device 30-day delay requirement. 

When an institution is required to 
prepare or to submit a certification 
with an application or proposal under 
these regulations, and the application 
or proposal involves an 
investigational new drug (within the 
meaning of 21 U.S.C. 355(i) or 
357(d)) or a significant risk device (as 
defined in 21 CFR 8l2.3(m)), the 
institution shall identify the drug or 
device in the certification. The 
institution shall also state whether the 
30-day interval required for 
investigational new drugs by 21 CFR 
312.1(a) and for significant risk 
devices by 21 CFR 812.30 has 
elapsed, or whether the Food and 
Drug Administration has waived that 
requirement. If the 30-day interval 
has expired, the institution shall state 
whether the Food and Drug 
Administration has requested that the 
sponsor continue to withhold or 
restrict the use of the drug or device 
in human subjects. If the 30-day 
interval has not expired, and a waiver 
has not been received, the institution 
shall send a statement to the 
Department upon expiration of the 
interval. The Department will not 
consider a certification acceptable 
until the institution has submitted a 
statement that the 30-day interval has 
elapsed, and the Food and Drug 
Administration has not requested it to 
limit the use of the drug or device, or 
that the Food and^ Drug 
Administration has waived the 30-day 
interval. 

§ 46.122 Use of Federal funds. 

Federal funds administered by the 
Department may not be expended for 
research involving human subjects 
unless the requirement of these 



regulations, including all subparts of 
these regulations* have been satisfied. 

5 46.123 Early termination of 
research funding; evaluation of 
subsequent applications and 
proposals. 

(a) The Secretary may require that 
Department funding for any project 
be terminated or suspended in the 
manner prescribed in applicable 
program requirements, when the 
Secretary finds an institution has 
materially failed to comply with the 
terms of these regulations. 

<b) In rr^king decisions about 
funding applicc:!or* or proposals 
covered by these regu'.iiions the 
Secretary may take into account, in 
addition to all other eligibility 
requirements and program criteria, 
factors such as whether the applicant 
has been subject to a termination or 
suspension under paragraph (a) of this 
section and whether the applicant or 
the person who would direct the 
scientific and technical aspects of an 
activity has in the judgment of the 
Secretary materially failed to 
discharge responsibility for the 
protection of the rights and welfare of 
human subjects (whether or not 
Department funds were invplved). 

8 46.124 Conditions. 

With respect to any research 
project or any class of research 
projects the Secretary may impose 
additional conditions prior to or at the 
time of funding when in the 
Secretary's judgment additional 
conditions are necessary for the 
protection of human subjects. 

Subpart B — Additional Protections 
Pertaining to Research 
Development , and Related 
Activities Involving Fetuses, 
Pregnant Women, and Human in 
Vitro Fertilization 

Source: 40 FR 33528. Aug. 8. 1975. 43 FR 
1758, January II, 1978.43 FR 
51559. November 3. 1978 

§ 46.201 Applicability. 

(a) The regulations in this subpart 
are applicable to all Department of 
Health, Education, and Welfare 
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grants and contract supporting 
research, development, and related 
activities involving: ( I) The fetus, (2) 
pregnant wome . and (3) human in 
vitro fertilization. 

(b) Nothing in ihis subpart shall be 
construed as indicating that 
compliance with the procedures set 
forth herein will in any wiy render 
inapplicable pertinent State or local 
laws bearing upon activities covered 
by this subpart. 

(c) The requirements of this 
subpart are in addition to those 
imposed under the other subparts of 
this part. 

i 46.202 Purpose. 

It is the purpose of this subpart to 
provide additional safeguards in 
reviewing activities to which this 
subpart is applicable to assure that 
they conform to appropriate ethical 
standards and relate to important 
societal needs. 

8 46.203 Definitions. 
As used in this subpart: 

(a) "Secretary" means the 
Secretary of Health, Education, and 
Welfare and any other officer or 
employee of the Department of 
Health. Education, and Welfare to 
whom authority has been delegated. 

(b) * 'Pregnancy" encompasses the 
period of time from confirmation of 
implantation (through any of the 
presumptive signs of pregnancy, such 
as missed menses, or by a medically 
acceptable pregnancy test), until 
expulsion or extraction of the fetus. 

(c) ••Fetus" means the product of 
conception from the time of 
implantation (as evidenced by any of 
the presumptive signs of pregnancy, 
such as missed menses, or a 
medically acceptable pregnancy test), 
until a determination is made, 
following expluston or extraction of 
the fetus, that it is viable. 

(d) M Viable" as it pertains to the 
fetus means being able, after either 
spontaneous or induced delivery, to 
survive (given the benefit of available 
medical therapy) to the point of 
independently maintaining heart 



beat and respiration* The Secretary 
m*y from time to time, taking into 
account medical advances, publish in 
the Federal Register guidelines 
to assist in determining whether a 
fetus is viable for purposes of i.:is 
subpart. If a fetus is viable after 
delivery, it is a premature infant. 

(e) "Nonviable fetus" means a 
fetus ex utero which, although living, 
is not viable. 

(0 "Dead fetus" means a fetus ex 
utero which exhibits neither 
heartbeat, spentant jus respiratory 
activity, spontaneous movement of 
voluntary muscles, iior pulsation of 
the umbilical cord (if still attached). 

(g) "In vitro fertilization" means 
any fertilization of human ova whi<,h 
occurs outside the body of a female, 
either through admixture of donor 
human sperm and ova or by any other 
means. 

5 46.204 Ethical Advisory 
Boards. 

(a) One or more Ethical Advisory 
Boards shall be established by the 
Secretary. Members of these board(s) 
shall be so selected that the board(s) 
will be competent to deal with 
medical, legal, social, ethical, and 
related issues and may include, for 
example, research scientists, 
physicians, psychologists, 
sociologists, educators, lawyers, and 
ethicists, as well as representatives of 
the general public. No board member 
may be a regular, full-time employee 
of the Department of Health, 
Edtcation, and Welfare. 

(b) At the request of the Secretary, 
the Ethical Advisory Board shall 
render advice consistent with the 
policies and requirements of this Part 
as to ethical issues, involving 
activities covered by this subpart, 
raised by individual applications or 
proposals. In addition, upon request 
by the Secretary, the Board shall 
render advice as to classes of 
applications or proposals and general 
policies, guidelines, and procedures. 

(c) A Board may establish, with 
the approval of the Secretary, classes 
of applications or proposals which: 



(I) Must be submitted to the Board, 
or (2) need not be submitted to the 
Board. Where the Board so 
establishes a class of applications or 
proposals which must be submitted, 
no application or proposal within the 
class may be funded by the 
Department or any component thereof 
until the application or proposal has 
been reviewed by the Board aad the 
Board has rendered advice as to its 
acceptability from an ethical 
standpoint. 

(d) No application or proposal 
involving human in vitro fertilization 
may be funded by the Department or 
any component thereof until the 
application or proposal has been 
reviewed by the Ethical Advisory 
Board and the Board has rendered 
advice as to its acceptability from an 
ethical standpoint. 

fi 46.205 Additional duties of the 
Institutional Review Boards In 
connection with activities 
involving fetuses, pregnant 
women, or human in vitro 
fertilization, 
(a) In addition to the 
responsibilities prescribed for 
Institutional Review Boards under 
Subpart A of this part, the applicant's 
or offeror's Board shall, with respect 
to activities covered by this subpart, 
carry out the following additional 
duties: 

(1) Determine that all aspects of 
the activity meet the requirements of 
this subpart; 

(2) Determine that adequate 
consideration has been given to the 
manner in which potential subjects 
will be selected, and adequate 
provision has been made by the 
applicant or offeror for monitoring 
the actual informed consent process 
(e.g., through such mechanisms, 
when appropriate, as participation by 
the Institutional Review Board or 
subject advocates in: (i) Overseeing 
the actual process by which 
individual consents required by this 
subpart are secured either by 
approving induction of each 
individual into the activity or 
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verifying, perhaps through sampling* 
that approved procedures for 
induction of individuals into the 
activity are being followed, and (ii) 
monitoring the progress of the 
activity and intervening as necessary 
through such steps as visits to the 
activity site and continuing evaluation 
to determine if any unanticipated 
risks have arisen); 

(3) Carry out such other 
responsibilities as may be assigned by 
the Secretary. 

(b) No award may be issued until 
the applicant or offeror has certified 
to the Secretary that the Institutional 
Review Board has made the 
determinations required under 
paragraph (a) of this section and the 
Secretary has approved these 
determinations, as provided in 

i 46.120 of Subpart A of this part. 

(c) Applicants or offerors seeking 
support for activities covered by this 
subpart must provide for the 
designation of an Institutional Review 
Board, subject to approval by th$ 
Secretary, where no such Board has 
been established under Subpart A of 
this part. 

146.206 General limitations. 

(a) No activity to which this 
subpart is applicable may be 
undertaken unless: 

(1) Appropriate studies on animals 
and nonpregnant individuals have 
been completed; 

(2) Except where the purpose of 
the activity is to meet the health 
needs of the mother or the particular 
fetus, the risk to the fetus is minimal 
and, in all cases, is the least possible 
risk for achieving the objectives of 
the activity. 

(3) Individuals engaged in the 
activity will have no part in: (i) Any 
decisions as to the timing, method, 
and procedures used to terminate the 
pregnancy, and (ii) determining the 
viability of the fetus at the 
termination of the pregnancy; and 

(4) No procedural changes which 
may cause greater than minimal risk 
to the fetus or the pregnant woman 
will be introduced into the procedure 



for terminating the pregnancy solely 
in the interest of the activity. 

(b) No inducements, monetary or 
otherwise, may be offered to 
terminate pregnancy for purposes of 
the activity. 

(40 FR 33528, Aug. I, 1973, as amended at 
40 FR 51638, Nov, 6, 19731 

5 46.207 Activities directed 
toward pregnant women as 
subjects. 

(a) No pregnant woman may be 
involved as a subject in an activity 
covered by this subpart unless: (I) 
The purpose of the activity is to meet 
the health needs of the mother and the 
fetus will be placed at risk only to the 
minimum extent necessary to meet 
such needs, or (2) the risk to the fetus 
is minimal. 

(b) An activity permitted under 
paragraph (a) of this section may be 
conducted only if the mother and 
father are legally competent and have 
given their informed consent after 
having been fully informed regarding 
possible impact on the fetus, except 
that the father's informed consent 
need not be secured if: ( I) The 
purpose of the activity is to meet the 
health needs of the mother; (2) his 
identity or whereabouts cannot 
reasonably be ascertained; (3) he is 
not reasonably available; or (4) the 
pregnancy resulted from rape. 

i 46.208 Activities directed 
toward fetuses In utero as 
subjects. 

(a) No fetus in utero may be 

involved as a subject in any activity 

covered by this subpart unless: (1) 

The purpose of the activity is to meet 

the health needs of the particular fetus 

and the fetus will be placed at risk 

only to the minimum extent necessary 

to meet such needs, or (2) the risk to 

the fetus imposed by the research is 

minimal and the purpose of the 

activity is the development of 

important biomedical knowledge 

which cannot be obtained by other 
means. 

(b) An activity permitted under 
paragraph (a) of this section may be 
conducted only if the mother and 

109 



father are legally competent and have 
given their informed consent, except 
that the father's consent need not be 
secured if: (I) His identity or 
whereabouts cannot reasonably be 
ascertained, (2) he is not reasonably 
available, or (3) the pregnancy 
resulted from rape. 

} 46.209 Activities directed 
toward fetuses ex utero, 
Inducting nonviable fetuses, as 
subjects. 

(a) Until it has been ascertained 
whether or not a fetus ex utero is 
viable, a fetus ex utero may not be 
involved as a subject ia an activity 
covered by this subpart unless: 

( 1) There will be no added risk to 
the fetus resulting from the activity, 
and the purpose of the activity is the 
development of important biomedical 
knowledge which rannot be obtained 
by other means, or 

(2) The purpose of the activity is to 
enhance the possibility of survival of 
the particular fetus to the point of 
viability. 

(b) No nonviable fetus may be 
involved as a subject in an activity 
covered by this subpart unless: 

(1) Vital functions of the fetus will 
not be artificially maintained, 

(2) Experimental activities which 
of themselves would terminate the 
heartbeat or respiration of the fetus 
will not be employed, and 

(3) The purpose of the activity is 
the development of important 
biomedical knowledge which cannot 
be obtained by other means. 

(c) In the event the fetus ex utero 
is found to be viable, it may be 
included as a subject in the activity 
only to the extent permitted by and in 
accordance with the requirements of 
other subparts of this part. 

(d) An activity permitted under 
paragraph (a) or (b) of this section 
may be conducted only if the mother 
and father are legally competent and 
have given their informed consent, 
except that the father's informed 
consent need not be secured if: (I) his 
identity or whereabouts cannot 
reasonably be ascertained, (2) he is 
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not reasonably available, or (3) the 
pregnancy resulted from rape. 

ft 46,210 Activities involving the 
dead fetus, fetal material 9 or the 
placenta* 

Activities involving the dead fetus, 
mascerated fetal material, or cells, 
tissue, or organs excised from a dead 
fetus shall be conducted only in 
accordance with any applicable State 
or local laws regarding such 
activities. 

fi 46.211 Modification or waiver 

of specific requirements. 

Upon the request of an applicant or 
offeror (with the approval of its 
Institutional Review Board), the 
Secretary may modify or waive 
specific requirements of this subpart* 
with the approval of the Ethical 
Advisory Board after such 
opportunity for public comment as 
the Ethical Advisory Board considers 
appropriate in the particular instance. 
In making such decisions, the 
Secretary will consider whether the 
risks to the subject are so outweighed 
by the sum of the benefit to the 
subject and the importance of the 
knowledge to be gained as to warrant 
such modification or waiver and that 
such benefits cannot be gained except 
through a modification or waiver. 
Any such modifications or waivers 
will be published as notice* in the 
Federal Register. 

Subpart C— Additional Protections 
Pertaining to Biomedical and 
Behavioral Research Involving 
Prisoners as Subjects 

Source: 43 FR 53655, Nov 16. 1978 

9 46.301 Applicability. 

(a) The regulations in this subpart 
are applicable to all biomedical and 
behavioral research conducted or 
supported by the Department of 
Health. Education, and Welfare 
involving prisoners as subjects. 

(b) Nothing in this subpart shall be 
construed as indicating that 
compliance with the procedures set 
forth herein will authorize research 
involving prisoners as subjects, to the 
extent such research is limited or 



barred by applicable State or local 
law. 

(c) The requirements of this 
subpart arc in addition to those 
imposed under the other subparts of 
this part. 

ft 46.302 Purpose. 

Inasmuch as prisoners ma" be 
under constraints because o. heir 
incarceration which could affect their 
ability to make a truly voluntary and 
uncoerced decision whether or not to 
participate as subjects in research, it 
is the purpose of this subpart to 
provide additional safeguards for the 
protection of prisoners involved in 
activities to which this subpart is 
applicable. 

ft 46.303 Definitions. 

As used in this subpart: 

(a) "Secretary" means the 
Secretary of Health, Education, and 
Welfare and any other officer or 
employee of the Department of 
Health. Education, and Welfare to 
whom authority has been delegated. 

(b) "DHEW" means the 
Department of Health. Education, 
and Welfare. 

(c) "Prisoner" means any 
individual involuntarily confined or 
detained in a penal institution. The 
term is intended to encompass 
individuals sentenced to such an 
institution under a criminal or civil 
statute, individuals detained in other 
facilities by virtue of statutes or 
commitment procedures which 
provide alternatives to criminal 
prosecution or incarceration in a 
penal institution, and individuals 
detained pending arraignment, trial, 
or sentencing. 

(d) "Minimal risk" is the 
probability and magnitude of physical 
or psychological harm that is 
normally encountered in the daily 
lives, or in the routine medical, 
dental, or psychological examination 
of healthy persons. 

ft 46.304 Composition of 
Institutional Review Boards 
where prisoners are involved. 
In addition to satisfying the 
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requirements in § 46.107 of this part, 
an Institutional Review Board, 
carrying out responsibilities under 
this part with respect to research 
covered by this subpart, shall also 
meet the following specific 
requirements: 

(a) A majority of the Board 
(exclusive of prisoner members) shall 
have no association with the prison(s) 
involved, apart from their 
membership on the Board. 

(b) At least one member of the 
Board shall be a prisoner, or a 
prisoner representative with 
appropriate background and 
experience to serve in that capacity, 
except that where a particular 
research project is reviewed by more 
than one Board only one Board need 
satisfy this requirement. 

ft 46.305 Additional duties of the 
Institutional Review Boards 
where prisoners are involved. 

(a) In addition to all other 
responsibilities prescribe for 
Institutional Review Boards under 
this part, the Board shall review 
research covered by this subpart and 
approve such research only if it finds 
that: 

(1) The research under review 
represents one of the categories of 
research permissible under 

§ 46.306(a)(2); 

(2) Any possible advantages 
accruing to the prisoner through his 
or her participation in the research, 
when compared to the general living 
conditions, medical care, quality of 
food, amenities and opportunity for 
earnings in the prison, are not of such 
a magnitude that his or her ability to 
weigh the risks of the research against 
the value of such advantages in the 
limited choice environment of the 
prison is impaired; 

(3) The risks involved in the 
research are commensurate with risks 
that would be accepted by 
nonprisoner volunteers; 

(4) Procedures for the selection of 
subjects within the prison are fair to 
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all prisoners and immune from 
arbitrary intervention by prison 
authorities or prisoners. Unless the 
principal investigator provides to the 
Board justification in writing for 
following some other procedures, 
control subjects must be selected 
randomly from the group of available 
prisoners who meet the characteristics 
needed for that particular research 
project; 

(5) The information is presented in 
language which is understandable to 
the subject population; 

(6) Adequate assurance exists that 
parole boards will not take into 
account a prisoner's participation in 
the research in making decisions 
regarding parole* and each prisoner is 
clearly informed in advance that 
participation in the research will have 
no effect on his or her parole; and 

(7) Where the Board finds there 
may be a need for follow-up 
examination or care of participants 
after. the end of their participation, 
adequate provision has been made for 
such examination or care, taking into 
account the varying lengths of 
individual prisoners' sentences, and 
for informing participants of this fact. 

(b) The Board shall carry out such 
other duties as may be assigned by 
the Secretary. 

(c) The institution shall certify to 
the Secretary, in such form and 
manner as the Secretary may require, 
that the duties of the Board under this 
section have been fulfilled. 

i 46*306 Permitted research 

involving prisoners. 

(a) Biomedical or behavioral 
research conducted or supported by 
DHEW may involve prisoners as 
subjects only if: 

(1) The institution responsible for 
the*conduct of the research has 
certified to the Secretary that »ie 
Institutional Review Board has 
approved the research under § 46.305 
of this subpart; and 

(2) In the judgment of .the 

O 
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Secretary the proposed research 
involves solely the following: 

(A) Study of the possible causes, 
effects, and processes of 
incarceration, and of criminal 
behavior, provided that the study 
presents no more than minimal risk 
and no more than inconvenience to 
the subjects; 

(B) Study of prisons as institutional 
structures or of prisoners as 
incarcerated persons, provided that 
the study presents no more than 
minimal risk and no more than 
inconvenience to the subjects; 

(C) Research on conditions 
particularly affecting prisoners as a 
class (for example, vaccine trials and 
other research on hepatitis which is 
much more prevalent in prisons than 
elsewhere; and research on social and 
psychological problems such as 
alcoholism, drug addiction and sexual 
assaults) provided that the study may 
proceed only after the Secretary has 
consulted with appropriate experts 
including experts in penology 
medicine and ethics, and published 
notice, in the Federal Register, 

of his intent to approve such research; 
or 

(D) Research on practices, both 
innovative and accepted, which have 
the intent and reasonable probability 
of improving the health or well- 
being of the subject. In cases in 
which those studies require the 
assignment of prisoners in a manner 
consistent with protocols approved by 
the IRB to control groups which may 
not benefit from the research, the 
study may proceed only after the 
Secretary has consulted with 
appropriate experts, including experts 
in penology medicine and ethics, and 
published notice, in the Federal 
Register, of hivintem to approve such 
research. 

(b) Except as provided in 
paragraph (a) of this section, 
biomedical or oehavioral research 
conducted or supported by DHEW 
shall not involve prisoners as 
subjects. 

in 



Subpart D— Additional Protections 
for Children Involved as Subjects in 
Research, 

Source: 48 FR 9818, Much 8, 1983 

§46.401 To what do these 
regulations apply? 

(a) This subpart applies to all 
research involving children as 
subjects, conducted or supported by 
the Department of Health and 
Human Services. 

(1) This includes research 
conducted by Department 
employees, except that each head of 
an Operating Division of the 
Department may adopt .such 
nonsubstantive, procedural 
modifications as may be appropriate 
from an administrative standpoint. 

(2) It also includes research 
conducted or supported by the 
Department of Health and Human 
Services outside the United States, 
but in appropriate circumstances, the 
Secretary may, under paragraph (e) 
of §46.101 of Subpart A, waive the 
applicability of some or all of the 
requirements of these regulations for 
research of this type. 

(b) Exemptions (1), (2), (5) and (6) 
as listed in Subpart A at §46. 101(b) 
are applicable to this subpart. 
Exemption (4), research involving 
the observation of public behavior, 
listed at §46, 101(b), is applicable to 
this subpart where the investigator(s) 
does not participate in the activities 
being observed. Exemption (3), 
research involving survey or 
interview procedures, listed at 

§ 46. 101(b) does not apply to research 
covered by this subpart. 

(c) The exceptions, additions, and 
provisions for waiver as they appear 
in paragraphs (c) through (i) of 
§46.101 of Subpart A are applicable 
to this subpart. 

§46.402 Definitions. 

The definitions in §46.102 of 
Subpart A shall be applicable to this 
subpart as well. In addition, as used 
in this subpart: 
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(a) "Children" are persons who 
have not attained the legal age for 
consent to treatments o» procedures 
involved in the research, under the 
applicable law of the jurisdiction in 
which the research will be 
conducted. 

(b) "Assent" means a child's 
affirmative agreement to participate 
in research. Mere failure to object 
should not, absent affirmative 
agreement, be construed as assent. 

(c) "Permission" means the 
agreement of parent(s) or guardian to 
the participation of their child or 
ward in research. 

(d) "Parent 91 means a child's 
biological or adoptive parent. 

(e) "Guardian" means an 
individual who is authorized under 
applicable state or local law to 
consent on behalf of a child to 
general medical care. 

§46.403 IRB duties. 

In addition to other responsibilities 
assigned to IRBs under this part, 
each IRB shall review research 
covered by this subpart and approve 
only research which satisfies the 
conditions of all applicable sections 
of this subpart. 

846.404 Research not involving 
greater than minimal risk. 

HHS will conduct or fund 
research in which the IRB finds that 
no greater than minimal risk to 
children is presented, only if the IRB 
finds that adequate provisions are 
made for soliciting the assent of the 
children and the permission of their 
parents or guardians, as set forth in 
1 46.408. 

§46.405 Research involving greater 
than minimal risk but presenting the 
prospect of direct benefit to the 
Individual subjects. 

HHS will conduct or fund 
research in which the IRB finds that 
more than minimal risk to children is 
presented by an intervention or 
procedure that holds out the 
prospect of direct benefit for the 
individual subject, or by a 



monitoring procedure that is likely to 
conthoute to the subject's well-being 
only if the IRB finds that: 

(a) The risk is justified by the 
anticipated benefit to the subjects; 

(b) The relation of the anticipated 
benefit to the risk is at least as 
favorable to the subjects as that 
presented by available alternative 
approaches; and 

(c) Adequate provisions are made 
for soliciting the assent of the 
children and permission of their 
parents or guardians, as set forth in 
§46.408. 

§46.406 Research involving greater 
than minimal risk and no prospect of 
direct benefit to individual subjects, 
but likely to yield generalizable 
knowledge about the subject's disorder 
or condition, 

HHS will conduct or fund 
research in which the IRB finds that 
more than minimal risk to children is 
presented by an intervention or 
procedure that does not hold out the 
prospect of direct benefit for the 
individual subject, or by a 
monitoring procedure which is not 
likely to contribute to the well-being 
of the subject, only if the IRB finds 
that: 

(a) The risk represents a minor 
increase over minimal risk; 

(b) The intervention or procedure 
presents experiences to subjects that 
are reasonably commensurate with 
those inherent in their actual or 
expected medical, dental, 
psychological, social, or educational 
situations; 

(c) The intervention or procedure 
is likely to yield generalizable 
knowledge about the subjects 1 
disorder or condition which is of 
vital importance for the 
understanding or amelioration of the 
subjects* disorder or condition; and 

(d) Adequate provisions are made 
for soliciting assent of the children 
and permission of their parents or 
guardians, as set forth in § 46.408. 
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$ 46.407 Research not otherwise 
approvable which presents air 
opportunity to understand, prevent, or 
alleviate a serious problem affecting 
the health or welfare of children. 

HHS will conduct or fund 
research that the IRB does not 
believe meets the requirements of 
S§ 46.404, 46.403, or 46.406 only if: 

(a) The IRB finds that the research 
presents a reasonable opportunity to 
further the understanding, 
prevention, or alleviation of a serious 
problem affecting the health or 
welfare of children; and 

(b) The Secretary, after 
consultation with a panel of experts 
in pertinent disciplines (for example: 
science, medicine, education, ethics, 
law) and following opportunity for 
public review and comment, has 
determined either: (I) That the 
research in fact satisfies the 
conditions of §§ 46.404, 46.405, or 
46.406, as applicable, or (2) the 
following: 

(i) The research presents a 
reasonable opportunity to further the 
understanding, prevention, or 
alleviation of a serious problem 
affecting the health or welfare of 
children; 

(ii) The research will be conducted 
in accordance with sound ethical 
principles; 

(iii) Adequate provisions are made 
for soliciting the assent of children 
and the permission of their parents or 
guardians, as set forth in § 46.408. 

§ 46.408 Requirements for 
permission by parents or guardians 
and for assent by children, 

(a) In addition to the 
determinations required under other 
applicable sections of this subpart, 
the IRB shall determine that 
adequate provisions are made for 
soliciting the assent of the children, 
when in the judgment of the IRB the 
children are capable of providing 
assent. In determining whether 
children are capable of assenting, the 
IRB shall take into account the ages, 
maturity, and psychological state of 
the children involved. This judgment 
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may be made for all children to be 
involved in research under a 
particular protocol, or for each child, 
as the IRB deems appropriate. If the 
IRB determines that the capability of 
some or all of the children is so 
limited that they cannot reasonably 
be consulted or that the intervention 
or procedure involved in the 
research holds out a prospect of 
direct benefit that is important to the 
health or well-being of the children 
and is available only in the context of 
the research, the assent of the 
children is not a necessary condition 
for proceeding with the research. 
Even where the IRB determines that 
the subjects are capable of assenting, 
the IRB may ;till waive the assent 
requirement under circumstances in 
which consent may be waived in 
accord with § 46.1 16 of Subpart A. 

(b) In addition to the 
determinations required under other 
applicable sections of this subpart, 
the IRB shall determine, in 
accordance with and to the extent 
that consent is required by § 46. 116 of 
Subpart A, that adequate provisions 
are made for soliciting the permission 
of each child's parents or guardian. 
Where parental permission is to be 
obtained, the IRB may find that the 
permission of one parent is sufficient 
for research to be conducted under 
§§ 46.404 or 46.405. Where research 
is covered by §§ 46.406 and 46.407 
and permission is to be obtained from 



parents, both parents must give their 
permission unless one parent is 
deceased, unknown, incompetent, or 
not reasonably available, or when 
only one parent has legal 
responsibility for the care and 
custody of the child. 

(c) In addition to the provisions for 
waiver contained in § 46.1 16 of 
Subpart A, if the IRB determines that 
a research protocol is designed for 
conditions or for a subject population 
for which parental or guardian 
permission is not a reasonable 
requirement to protect the subjects 
(for example, neglected or abused 
children), it may waive the consent 
requirements in Subpart A of this 
part and paragraph (b) of this section, 
provided an appropriate mechanism 
for protecting the children who will 
participate as subjects in the research 
is substituted, and provided further 
that the waiver is not inconsistent 
with federal state or local law. The 
choice of an appropriate mechanism 
would depend upon the nature and 
purpose of the activities described in 
the protocol, the risk and anticipated 
benefit to the research subjects, and 
their age, maturity, status, and 
condition. 

(d) Permission by parents or 
guardians shall be documented in 
accordance with and to the extent 
required by $ 46.1 17 of Subpart A. 

(e) When the IRB determines that 
assent is required, it shall also 
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determine whether and how assent 
must be documented. 



$46,409 Wards. 

(a) Children who are wards of the 
state or any other agency, institution, 
or entity can be included in research 
approved under $8 4*406 or 46.407 
only if such research is: 

(1) Related to their status as wards; 

or 

(2) Conducted in schools, camps, 
hospitals, institutions, or similar 
settings in which the majority of 
children involved as subjects are not 
wards. 

(b) If the research is approved 
under paragraph (a) of this section, 
the IRB shall require appointment of 
an advocate for each child who is a 
ward, in addition to any other 
individual acting on behalf of the 
child as guardian or in loco parentis. 
One individual may serve as 
advocate for more than one child. 
The advocate shall be an individual 
who has the background and 
experience to act in, and agrees to act 
in, the best interests of the child for 
the duration of the child's 
participation in the research and who 
is not associated in any way-(except 
in the role as advocate or member of 
the IRB) with the research, the 
investigator(s), or the guardian 
organization. 
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NOTICES 



HUMAN SUBJECTS 
Minimum Criteria Identifying the 
Viable Fetus 

On March 13, 1975, regulations 
were published in che Federal 
Register (40 FR 1 1854) relating to the 
protection of human subjects in 
research, development, and related 
activities supported by Department of 
Health, Education, and Welfare 
grants and contracts. These 
regulations are codified at 45 CFR 
Part 46. 



Elsewhere in this issue of the 
Federal Register, the Secretary 
is amending 45 CFR Part 46 by,- 
among other things, adding a new 
Subpart B to provide additional 
protections pertaining to research, 
development, and related activities 
involving fetuses, pregnant women, 
and in vitro fertilization. 

Section 46.203(d) of Subpart B 
provides inter alia as follows: 

The Secretary may from lime to time, 
taking into account medical advances, 
publish in the Federal Register 



guidelines to assist in determiniif whether a 
fetus is viable for purposes of this subpart. 

This notice is published in 
accordance with $ 46.203(d). For 
purposes of Subpart B, the guidelines 
indicating that a fetus other than a 
dead fetus within the meaning of 
§ 46.203(0 is viable include the 
following: 

an estimated gestational age of 20 weeks or 
more and a body weight of 300 (rant or 

more. 

FEDERAL REGISTER, VOL 40, 
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RESEARCH ACTIVITIES WHICH MAY BE REVIEWED 
THROUGH EXPEDITED REVIEW PROCEDURES 



Research activities involving no more than minimal risk and in which the 
only involvement of human subjects will be in one or more of the 
following categories (carried out through standard methods) may be 
reviewed by the Institutional Review Board through the txpedited review 
procedure authorized in 46.110 of 45 CFR Part 46. 



(1) Collection of: hair and 
nail clippings, in a 
nondisf iguring manner; 
deciduous teeth; and permanent 
teeth if patient care indicates 
a need for extraction. 

(2) Collection of excreta 
and external secretions 
including sweat, uncannulated 
saliva, placenta removed at 
delivery, and amniotic fluid at 
the time of rupture of the 
membrane prior to or during 
labor. 

(3) Recording of data from 
subjects 18 years of age or 
older using noninvasive 
procedures routinely employed 
in clinical practice. This 
includes the use of physical 
sensors that are applied either 
to the surface of the body or 
at a distance and do not 
involve input of matter or 
significant amounts of energy 
into the subject or an invasion 
of the subject's privacy. It 
also includes such procedures 
as weighing, testing sensory 
acuity, electrocardiography, 
electroencephalography , 
thermography, detection of 
naturally occurring 
radioactivity, diagnostic 
echography, and 
electroretinography . It does 
not include exposure to 
electromagnetic radiation 
outside the visible range (for 
example, x-rays, microwaves). 

(4) Collection of blood 
samples by venipuncture, in 
amounts not exceeding 450 

• U.S..C0VEW.MENT PRINTING OfflCEi IM-*1 I- I >!»*»*» 



milliliters in an eight-week 
period and no more often than 
two times per week, from 
subjects 18 years of age or 
older and who are in good 
health and not pregnant. 

(5) Collection of both 
supra- and subgingival dental 
plaque and calculus, provided 
the procedure is not more 
invasive than routine 
prophylactic scaling of the 
teeth and the process is 
accomplished in accordance with 
accepted prophylactic 
techniques. 

(6) Voice recordings made 
for research purposes such as 
investigations of speech 
defects. 

(7) Moderate exercise by 
healthy volunteers. 

(8) The study of existing 
data, documents, records, 
pathological specimens, or 
diagnostic specimens. 

(9) Research on individual 
or group behavior or 
characteristics of individuals, 
such as studies of perception, 
cognition, game theory, or test 
development, where the 
investigator does not 
manipulate subjects' behavior 
and the research will not 
involve stress to subjects. 

(10) Research on drugs or 
devices for which an 
investigational new drug 
exemption or an investigational 
device exemption is not 
required. 
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TECHNICAL ASSISTANCE RESOURCES 
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TECHNICAL ASSISTANCE RESOURCES 



Child and Adolescent Service System Program (CASSP) 

Child and Family Support Branch 

Division of Applied and Services Research 

Room 11 C-09 

5600 Fishers L ie 

Rockville.MD 20857 

(301)443-1333 



CASSP Technical Assistance Center 
Georgetown University Child Development Center 
2233 Wisconsin Avenue, N.W. 
Washington, D.C 20007 
(202)338-1831 



Research and Training Center for 
Children's Mental Health 
Florida Mental Health Institute 
University of South Florida 
13301 Bruce B. Downs Blvd. 
Tampa, FL 33612-3899 
(813)9744500 



Research and Training Center on 

Family Support and Children's Mental Health 

Portland State University 

P.O. Box 751 

Portland, OR 97207-0751 

(503)725-4040 
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